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ABSTRACT 


National- and Department-level deeision-makers expeet eredible Department of 
Defense models and simulations (M&S) to provide them eonfidence in the simulation 
results, espeeially for mission-eritieal and high-risk deeisions supporting National Secu¬ 
rity. Many of these large-scale, software-intensive simulation systems were autono¬ 
mously developed over time, and subject to varying degrees of funding, maintenance, and 
life-cycle management practices, resulting in heterogeneous model representations and 
data. Systemic problems with distributed interoperability of these non-trivial simulations 
in federations’ persist, and current techniques, procedures, and tools have not achieved 
the desired results. 

The Software Architecture-Based Product Line for simulation model representa¬ 
tions, employing Architecture Readiness Levels presented in this dissertation provides an 
alternative methodology. The proposed four-layered M&S software architecture-based 
product line model enables the development of model representations supported by 
readiness levels. Each layer reflects a division of the software architecture-based product 
line. The layer represents a horizontal slice through the architecture for organizing view¬ 
points or views at the same level of abstraction while the software architecture-based 
product line represents a vertical slice. A layer may maintain multiple views and view¬ 
points of a software architecture-based product line. A Domain Metadata Repository 
prescribes the interaction between layers. We introduce the Domain Integrated Product 
Development Team concept. 
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I. 


INTRODUCTION 


A. THE NEW STRATEGIC ROLE FOR MODELS AND SIMULATIONS 

Software' enables the politieal [Bus02d], eeonomie [RTI02], psyehologieal 
[Bus02b, Bus02e], and military [RumOl] eenters of United States national power. The 

2 

United States military center of national power, manifested in the Department of Defense 

-5 

(DoD), employs software-intensive systems in a wide variety of missions, tasks, and func¬ 
tions, including models and simulations"' (M&S), supporting the Department’s national de¬ 
fense mission. The Department requires credible^ M&S supporting confidence^ in the 
simulation results provided to Department- and National-level decision-makers, especially 
in high-risk, mission-critical decisions [DoD94]. Establishing credibility in Department 
simulations involves many disciplines and knowledge areas including software engineer¬ 
ing’, processes^, quality^, product management"* and architecture". The Department’s 
complex organizational dynamics, and complicated acquisition procedures [BSK+OO, 
SAG+00, Wol02a, Wol02b] also impact the level of M&S credibility, at times adversely. 

There is a new sense of urgency in the Department for credible M&S, and an in¬ 
creased emphasis on improved confidence in simulation results [DoDOSa]. In the past, 
models and simulations played secondary and tertiary roles supporting the Department’s 
decision-making process. Today, models and simulations are Department corporate assets 
[DoD94], evolving into primary enablers directly supporting an information age national 

^ Software includes the computer programs, procedures, and possibly associated documentation and data pertaining to the operation of a 

computer system [IEE90]. 

2 

Department of Defense will be referred to by the terms Department or DoD, used interchangeably throughout this document. 

3 

A software-intensive system is any system where software contributes essential influences to the design, construction, deployment, and 

evolution of the system as a whole [lEEOOb]. 

4 

Models and simulations (M&S) - In this research models and simulations (M&S) refers to mathematical abstractions and software 
implementations, as opposed to the term modeling and simulations used to identify an analytical problem solving approach [GMS+96]. 

Credible means 1. Capable of being believed; believable; plausible; 2. Worthy of confidence; reliable. Plausible is defined as 1. Seem¬ 
ingly or apparently valid, likely, or acceptable. Reliable is further defined as 1. That can be relied upon; dependable [Web95]. 

^ Confidence is defined as 1.Trust or faith. 2. A trusting relationship. 3. Something confided. 4. A feeling of assurance, especially of self- 
assurance. 5. The assurance that someone keeps a secret [Web95]. 

7 

Software engineering is the application of a systematic, disciplined, quantifiable approach to the development; operation, and mainte¬ 
nance of software; that is, the application of engineering to software [IEE90]. 

Process is a sequence of steps performed for a given purpose: for example, the software, development process [IEE90]. 

9 

Quality is 1). The degree to which a system, component, or process meets specified requirements; 2). The degree to which a system, 
component, or process meets customer or user needs and expectations [IEE90]. 

Product management includes the definition, coordination, and control of the product characteristics during its development cycle 
[IEE90. 

Architecture means the organizational structure of a system or component [IEE90]. 

1 



security strategy [Bus02d]. The Department’s support for this strategy and supporting ob¬ 
jectives requires confidence, credibility, security and interoperability in M&S, which are 
“broadly applicable to the full range and scope of missions and activities across the breadth 
of DoD operations” [DoDOla] and enable the Department’s execution of critical missions, 
tasks, and functions [Sha97]. 

National-level decision-makers require credible mission-critical M&S software^^ 
supporting confidence in national security policy decisions including the emerging Home¬ 
land Defense security policy [FOP+01, Bre02b, Bus02a, Bus02b, Bus02c, CS02] and com¬ 
plex, mission-critical, high-risk, defensive systems such as ballistic missile defense, de¬ 
signed to counter asymmetric twenty-first century threats against the American Homeland, 
including global terrorism and weapons of mass destruction [Bus02e]. President Bush re¬ 
cently signed the National Strategy For Homeland Security [Bus02c] emphasizing six mis¬ 
sion-critical areas supporting the Nation’s defense including the protection of critical infra¬ 
structure and key assets. Within the critical infrastructure mission area, the President iden¬ 
tified eight major initiatives including the development of a national infrastructure protec¬ 
tion plan, securing cyberspace, and harnessing “the best analytic and modeling tools to de¬ 
velop effective protective solutions” [Bus02c]. 

From a strategic perspective, [DoD02d] noted, “Modeling and simulation are sim¬ 
ply a means to an end” [DoD02d]. Confidence in the high-risk, mission-critical results pro¬ 
duced from the Department’s simulations employed as a means for National Security 
necessitate the highest levels of credibility. The Department’s dva&. Modeling and Simula¬ 
tion Strategic Plan [DoDOSa] reflects this new emphasis on the warfighter with the first 
Department M&S mission statement: “Exploit Modeling and Simulation to Transform the 
Warfighting Capabilities of the United States” [DoDOSa]. 
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Critical software is software whose failure could have on safety, or could cause large financial or social loss [IEE90]. 
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AUTHORITATIVE REPRESENTATIONS AND CREDIBLE M&S 
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Credible M&S requires authoritative representations , aeeording to the Depart¬ 
ment’s M&S poliey direetives, DoD Modeling and Simulation Management [DoD94] and 
the Department’s Modeling and Simulation Master Plan [DoD95]. A Department M&S 
baseline assessment [DoD95] identified authoritative representations as one of the “many 
shortfalls that must be eorreeted to realize the DoD M&S vision”^"^ [DoD95] and support 
the Department’s readiness, modernization, foree strueture, and sustainability requirements. 
In this researeh we employed the coneept of eomputer-based simulations defined by 
[CS97] as a wide range of eomputationally based activities employing digital computers. 

In addition, we view the Department’s large-scale, legacy'^ modeling and simulation sys¬ 
tems as software-intensive systems. 

Attesting to the importance of credible M&S representations in Department M&S, 
three of the six major [DoD95] objectives addressed authoritative representations. This 
research focused on [DoD95] objective three, authoritative system^^ representations. Other 
major [DoD95] objectives included the natural environment authoritative representations 
(objective two), and human behavior authoritative representations (objective four). Ena¬ 
bling tasks and timelines established to achieve this [DOD95] objective included: 

• Build prototype classes of objects as part of the architecture prototype effort in 
fiscal year 1995, 

• Identify common object classes in fiscal year 1996, 

• Assign DoD executive agents to develop common object classes in fiscal year 1996, 

• Identify and assign responsibilities for developing system M&S for all DoD mission 
areas by fiscal year 1997, 


Authoritative representations are models, algorithms, and data that have been developed or approved by a source which have accurate 

technical knowledge of the entity or phenomenon to be modeled and its effects [DoD95]. 

14 

Department M&S Vision (1995) A. Defense modeling and simulation will provide readily available, operationally valid environments 
for use by the DoD Components: (1) To train jointly, develop doctrine and tactics, formulate operational plans, and assess warfighting 
situations. (2) To support technology assessment, system upgrade, prototype and full-scale development, and force structuring. B. Fur¬ 
thermore, common use of these environments will promote a closer interaction between the operations and acquisition community in 
carrying out their respective duties. To allow maximum utility and flexibility, these modeling and simulation environments will be con¬ 
structed from affordable, reusable components interoperating through an open-system architecture [DoD95]. 

Legacy M&S - M&S systems developed in the past without the benefit of modem standards, and still in use [PMR94]. 

DoD M&S system representations include weapons, sensors, units, command, control, communications, computers and intelligence, 
surveillance, reconnaissance (C4ISR) platforms, and logistic support for a wide scope of U.S., Allied, Coalition, and major threat plat¬ 
forms [DoD95]. 

17 

Architecture in this context is defined as the stmcture of components in a program/system, their interrelationships, and principles and 
guidelines governing their design and evolution over time [DoD95]. Other definitions are introduced later. 

3 
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• Develop aggregation/disaggregation methodologies by a date to be determined 
[DoD95]. 

C. LIMITATIONS OF CURRENT APPROACHES TO ACHIEVE CREDIBIL¬ 
ITY 

The Defense Modeling and Simulation Offiee (DMSO) initiated several studies in 
2000 to assess the progress of M&S within the Department sinee the 1995 publieation of 
the M&S Master Plan [DoD95]. The DMSO methodology employed multiple techniques 
including written surveys and interviews [WriOlb] with key members of the Department’s 
M&S community, sponsorship of the Integration Task Force (ITF) effort, and completion 
of a Warfighter M&S needs assessment. These initiatives produced three major product 
deliverables, 2 l Master Plan Baseline Assessment [MosOO, CraOla, CraOlb], a final assess¬ 
ment Report of Warfighter M&S Needs [MWD+OO], and the Report of the Integration Task 
Force [HCG+Ola, HCB+Olb]. The survey results indicated that most of the [DoD95] ob¬ 
jectives experienced some progress, with the greatest progress noted on the development of 
a common technical framework, and the establishment of the High Level Architecture 
(HLA), an Institute of Electrical and Electronic Engineers (IEEE) standard [IEE98e, 
lEEOOc, lEEOOd, lEEOOe], and the Department’s technical architecture for simulation in¬ 
teroperability [GanOO]. 

However, most objectives for authoritative representations remained incomplete, 
with the least progress noted on the representation of human behavior [MosOO, MTOl, 
Elo02a]. The three DMSO study initiatives [HCG+Ola, HCG+Olb, MWD+OO, and MosOO] 
collectively provided direction to the Department’s 2001 draft revision of the DoD Model¬ 
ing and Simulation Master Plan (Draft) [DoDOla]. Many of the persistent systemic prob¬ 
lems with authoritative representations require long-term executable solutions to improve 
quality and evidence that M&S provides a return on investment [Mye99]. The true scope 
of the issues, challenges, and difficulties faced by the Department’s M&S community to 
develop authoritative representations is evident by a comparison of the system authoritative 
representation sub-objectives of the approved [DoD95] and the draft [DoDOla] plan. Simi- 

Aggregation is the ability to group entities while preserving the effects of entity behavior and interaction while grouped. Disaggrega¬ 
tion is the ability to represent the behavior of an aggregated unit in terms of its component entities based on maintained state representa¬ 
tions [DoD98]. 
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lar [DoD95] objectives originally scheduled for implementation in the 1995-1997 time- 
frame, slipped almost a decade to 2004-2006 [DoDOla], 

There were several understandable reasons for the [DoD95] schedule slip. A few of 
the reasons included a very ambitious schedule negatively impacted by Department budget 
cuts during the 1990s; the sheer scope of the work across the Department, Service and 
M&S domains^^; the breadth of the technical challenges, including improved fidelity^'^ of 
authoritative representations [Gro99, RGHOO]; and the overall immaturity of software en¬ 
gineering processes and practices [HHK93, HNC+00] supporting M&S development. 

Other contributing factors to the [DoD95] schedule slip included significant interoperabil¬ 
ity issues, external forces including the Department’s balky acquisition process [Mos02], 
changing Department priorities [FerOl]; and the unplanned allocation of resources to Year 
2000 problem remediation [Gre97, WG99]. The draft [DoDOla] again reiterated the need 
for “credible authoritative representations” (sub-objective 3.3), which are interoperable, 
consistent, and promote “access to the underlying algorithms and data, which define the 
representation within M&S by other applications” [DoDOla]. 

The Department also continues to experience persistent, systemic M&S architecture 
problems. The C4ISR Information Superiority Master Plan [DoD02d] identified continu¬ 
ing concerns with the Department’s M&S technical architecture noting the HLA for De¬ 
partment M&S lacked seamless interoperability with the Common Operating Environment 
(COE) [JTA02a, JTA02b] supporting the development of the Global Command and Con¬ 
trol System (GCCS) and the Department’s Command, Control, Communications, Com¬ 
puters, Intelligence, Surveillance and Reconnaissance (C4ISR) architecture framework 
[AWG97]. The proposed [DoD02d] solution is to develop a common Technical Reference 
Model (TRM) for the C4ISR and the M&S community, evolving over the next twenty 


19 

Domains are the physical or abstract space in which the entities and processes operate. The domain can be the land, sea, air, space, 
undersea, a combination of any these, or an abstract domain such as an n-dimensional mathematics space, or economic or psychological 
domains [DoD98]. 

20 

Fidelity: 1. The degree to which a model or simulation reproduces the state and behavior of a real world object or the perception of a 
real world object, feature, condition, or chosen standard in a measurable or perceivable manner; a measure of the realism of a model or 
simulation; faithfulness. Fidelity should generally be described with respect to the measures, standards or perceptions used in assessing 
or stating it. 2. The methods, metrics, and descriptions of models or simulations used to compare those models or simulations to their 
real world referents or to other simulations in such terms as accuracy, scope, resolution, level of detail, level of abstraction and repeat¬ 
ability. Fidelity can characterize the representations of a model, a simulation, the data used by a simulation (e.g., input, characteristic or 
parametric), or an exercise. Each of these fidelity types has different implications for the applications that employ these representations 
[Gro99, RGHOO, RPGOO] 
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years. Based on the aetual eompletion dates of previous Department M&S projeets, this 
time-to-eompletion estimate for a TRM may be accurate. However, it is not clear if this 
timeline is responsive to the Department’s operational requirements and transformation ini¬ 
tiatives [MAKOl], since the goal for the Department is to have legacy systems interoper¬ 
able, in terms of command and control functions, by the end of fiscal year 2008 [WolOl, 
FBK+01]. 

Lessons-leamed from incorporating M&S into the Department’s test and evaluation 
(T&E) process [DOTOla, DOTOlb] confirmed the inherent limits of M&S cited twenty 
years earlier [PAD78, PAD80]. [DOTOla, DOTOlb] revalidated the lack of quality, au¬ 
thoritative system representations as the Department’s chronic systemic M&S shortfall 
with the “underlying issues of the credibility of the tools [being] a major constraint” 
[PHP+96]. Improving the fidelity , accuracy and precision quality attributes of au¬ 
thoritative representations supports credibility. However, M&S quality has also been an 
elusive objective for the Department. Many prior systemic deficiencies continue to exist, 
including the need for authoritative representations and data, improved information assur¬ 
ance measures [HGE03], and interoperability standards [HCG+Ola, HCG+Olb, You02c]. 
We refer to the entire class of problems impeding the development of authoritative repre¬ 
sentations as heterogeneous system representation anomalies [IEE93], discussed in Chapter 
IV. 

Heterogeneous system representation anomalies, a major focus of this study, in¬ 
clude two broad categories adversely affecting simulation federation credibility: technical 
interoperability and substantive interoperability [DSB+99, YPHOl]. Technical interopera¬ 
bility [YPHOl] is the capability of federates to physically connect and exchange data, while 
substantive interoperability [YPHOl] adds the requirement for the federation to provide 


Fidelity is the accuracy of the representation when compared to the real world [DoD95]. 

22 

Accuracy: The degree to which a parameter or variable or set of parameters or variables within a model or simulation conform ex¬ 
actly to reality or to some chosen standard or referent. See resolution, fidelity, precision [RPGOO]. 

23 

Precision: 1. The quality or state of being clearly depicted, definite, measured or calculated. 2. A quality associated with the spread 
of data obtained in repetitions of an experiment as measured by variance; the lower the variance, the higher the precision. 3. A measure 
of how meticulously or rigorously computational processes are described or performed by a model or simulation [RPGOO]. 

24 

An attribute is a property or characteristic of one or more entities, a property inherent in an entity, or associated with that entity for 
database purposes [DoD98]. 
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consistent, accurate simulated representations, adhering to “fair fight” principles'^ to ade¬ 
quately meet federation accreditation (e.g., intended use) objectives. Substantive interop¬ 
erability [Dav98, DSB-l-99, YPHOl] involves the representation validity supporting the fed- 
erations’ intended use, interoperability among entities , and the eontextual effects on in- 

97 

teroperability . The logical interaction of federation entities involves: 

• The level of representation for the entity to include the neeessary level of detail and 
goodness-of-fit, 

• The key entity model attributes refleeting eritieal real-world eharacteristics salient 
to the federation’s purpose, 

• The entity models exhibit the needed behaviors, work together logically, and em¬ 
ploy consistent algorithms aeross the federation to compute effeets [YPHOl]. 

D. RESEARCH QUESTION AND HYPOTHESIS 

The primary foeus of this dissertation was to determine if system representations in 
five selected Missile Defense Agency’s (MDA) software-intensive M&S systems were 
authoritative according to current Department standards, and credible enough to support 
eonfidence by Department- and National-level deeision-makers in missile defense deci¬ 
sions critical to national security. 

The second objective included the identifieation of pre-eonditions and alternatives 
to improve and enhanee the simulation credibility and confidenee in the results produced 
by the Agency’s M&S software. The basie research problem investigated during this effort 
was: 

How may the Ageney better define, develop, evolve, manage, and maintain authori¬ 
tative model representations in the five selected large-scale, software-intensive legacy 
simulations , and effectively address simulation interoperability issues (e.g., heterogene¬ 
ous system representation anomalies) within existing constraints and pre-conditions (e.g.. 


25 

Two or more simulations may be considered to be in a fair fight when differences’ in the simulations’ performance characteristics 
have significantly less effect on the outcome of the conflict than actions taken by the simulation participants [DoD98]. 

Interoperability among entities includes accurate temporal resolution, spatial resolution and the logical interaction among entities 

modeled in different federates [YPHOl]. 

27 

Contextual effects on interoperability address the accurate, coherent relationship required between components of the physical envi¬ 
ronment to include ocean, ground, space, atmosphere, weather, infrared and electromagnetic spectrums [HYOlaj. 

Hereafter, the Missile Defense Agency maybe referred to as the Agency or the Ballistic Missile Defense System (BMDS). 

29 

Chapter IX provides the analysis of the five selected Agency simulations under study. 
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technical, organizational, managerial) affecting credibility in the five selected simulations 
systems? 

In response to this question I submit the following hypothesis: Employing a do- 

30 31 32 

main -managed, four-layered simulation software architecture-based , product line 
model with a referent layer, developed in a distributed domain integrated software engi¬ 
neering environment supporting the evolution of five selected Agency legacy simulations 
can improve the authority of representations in the five selected missile defense simula¬ 
tions. The Software Architecture-Based Product Line Model developed in a distributed 
development environment by a Domain Integrated Product Development Team employing 
Architecture Readiness Levels (ARL) to measure quality provides such a methodology. 

The methodology includes software development practice areas [JS02]: software engineer- 
ing , organizational management and technical management [CN02, Nor03] compo¬ 
nents of the Department’s system engineering process [DAUOI]. Hypothesis testing in¬ 
cludes concurrent implementation of the techniques and procedures in the Agency’s M&S 
domain. 

E, SIMULATION SOFTWARE ARCHITECTURE (SSA) OVERVIEW 

This dissertation introduces a four-layer, simulation software architecture-based 
product line model, and architecture readiness levels supporting distributed simulation 
software development by a domain integrated product development teams employing prod- 


30 

Domain for the purpose of this research is an area of knowledge or activity characterized by a set of concepts and terminology under¬ 
stood by practitioners in that area [Nor03]. A more M&S domain-unique definition for domain is provided in [DoD98]. 

31 

Software architecture is the fundamental organization of a system, embodied in its components, their relationships to each other and 

the environment, and the principles governing its design and evolution [lEEOOb]. 

32 

A product line is a group of products sharing a common, managed set of features that satisfy specific needs of selected market or 

mission [BFG+OO], 

33 

1. A referent is a codified body of knowledge about a thing being simulated [RPGOO]. 2. Something referenced or singled out for 

attention, a designated object, real or imaginary or any class of such objects [Gro99, RGHOO]. 

34 

Software engineering practice areas are those practices necessary to apply the appropriate technology to create and evolve both core 

assets and products [JS02]. 

35 

Organizational management refers to the management of the business issues that are visible at the enterprise level, as opposed to the 
project level [JS02]. 

Technical management practices areas include those management practices necessary to engineer the development and evolution of 

core assets and products [JS02]. 

37 

Architecture-based development is a process that utilizes the software architecture as the primary tool for the design, evolution, im¬ 
plementation, management, migration, and understanding of a software system. It involves organizing the work products around the 
architecture; implementing the software system based on the architecture; and maintaining the implementation to reflect changes in, and 
to ensure conformance to, the architecture [CN96]. 


8 



uct line practice areas to evolve Agency’s the five legacy simulations heterogeneously or- 
ganized in a joint M&S hierarchy today. The proposed four-layered simulation soft¬ 
ware architecture-based product line model supported by a domain metadata repository en¬ 
ables the development of authoritative representations supported by readiness levels"^'^. 

The four-level layered software architecture-based product line model replaces the 
level of abstraction"^^ prescribed previously in the traditional Department M&S hierarchy"^^ 
(Figure 2-2), with each layer (e.g., referent, conceptual, component, implementation) re¬ 
flecting a division of the software architecture-based product line model into units. The 
layer represents a horizontal slice through the architecture for organizing classifiers or 
packages at the same level of abstraction [UML99, MB02, Mil02b], while the software ar¬ 
chitecture-based product line represents a vertical slice. Vertical slice components of the 
model include a Simulation Software Architectural Framework (SSAF) and the Simulation 
Product Line Architecture (SPLA). A layering scheme prescribes the interaction between 
layers. A Referent layer maintains multiple real-world views"^^ and viewpoints"^"^ for estab¬ 
lishing the foundation of authoritative model representations. 

The methodology supports development of software architecture-based product line 
readiness levels or Architecture Readiness Levels (ARL) and a domain metadata reposi¬ 
tory, addressed in Chapter X. The ARL methodology supports conceptual model valida¬ 
tion, data management, the verification, validation process, and the implementation of qual¬ 
ity attributes instantiated in the software implementation. The ARL methodology also sup- 


Joint M&S includes representations of joint and service forces, capabilities, equipment, materiel, and services used by the joint com¬ 
munity or by two, or more, Military Services [DoD95]. 

39 

Hierarchy is a ranking or ordering of abstractions [DoD98]. 

40 

Readiness Levels are based on Technology Readiness Levels - a systemic metric/measurement system developed by NASA that sup¬ 
ports assessments of the maturity of a particular technology and the consistent comparison between different types of technology 
[Man95]. 

41 

Abstraction is a view of an object that focuses in the information relevant to a particular purpose and ignores the reminder of the in¬ 
formation [IEE90]. 

We address simulation hierarchies in Chapter IL Originally devised to manage simulation complexity, multiple viewpoints of simu¬ 
lation hierarchies evolved in many forms. The Department selected a version of the hierarchy cited in Figure 2-2. Systemic issues per¬ 
sist with this simulation hierarchy model, and the lack of integration between the different layers of the simulation hierarchy. [WGB+03] 

most recently addressed this systemic shortcoming of the Department’s M&S architecture. 

43 

In the conceptual framework of IEEE Std 1471-2000, an architectural description (AD) is organized into one or more constituents 
identified as architectural views. A view addresses one or more concerns identified by system stakeholders. The term view is used to 
refer to the expression of the systems architecture with respect to a particular viewpoint. A view may consist of one or more architectural 

model [lEEOOb]. 

44 

The viewpoint is a specification of the conventions for constructing and using a view, a pattern or template from which to develop 
individual views by establishing the purposes and audience for a view and the techniques for its creation and analysis. The viewpoint 
determines the language (including notation, model, or product types) to be used to describe the view, and any associated modeling 
methods or analysis techniques applied to representations of the view [lEEOOb]. 
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ports the development of quantitative assessments of system representations by the devel¬ 
oper and enables the measurement of the produetion proeess. The domain metadata reposi¬ 
tory supports the domain metadata registry supporting interoperability with other Govern¬ 
ment agencies and private sector metadata registries. 

The centrally-managed, distributed, domain integrated product development team 
supports the second study objective: a distributed M&S software engineering management 
methodology integrating the “production-related” approach to attain high levels of quality 
by defining specific characteristics of a product with precise measurements, supported by 
the “process-related”"^^ approach to prevent faults and minimize rework. The production 
related approach employs a distributed architecture-based software development model 
based on product lines. 

The target audiences for the software architecture-based product line family include 
Departmental- and National-level decision-makers. Other key stakeholders supported by 
this concept include the Department’s M&S community; simulation software developers; 
supported analysis, training, acquisition, experimentation, and operational communities; 
developmental and operational testers; and the private sector supporting M&S software en¬ 
gineering, transformation initiatives, and simulation-based acquisition (SBA). 

F, RESEARCH CONTRIBUTIONS 

The Department seeks improved credibility in M&S supporting Departmental- and 
National-level decision-makers’ confidence in simulation results. Major issues with the 
implementation of the Department’s verification, validation, and accreditation (VV&A) 
process, conceptual model development practices, software development maturity, process 
improvement, and M&S quality have caused Department decision-makers to question the 
amount of confidence they should place in results produced by the Department’s M&S. As 
a result is has been difficult for the Department to identify and justify a return on invest- 


45 

Acquisition of software-intensive systems shall use process improvement and performance measures [DoD03b]. 
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ment for M&S, especially for large-scale M&S costing hundreds of million dollars or even 
exceeding a billion dollars"^^, to develop and maintain [BGK+OO]. 

Conversely, the demand for improved M&S capability, interoperability, reusability, 
and affordability continue to grow [DoD03a], driven by the need to transform United States 
warfighting capability. The main contributions of this work add to the model and simula¬ 
tion body of knowledge and address a causal factor of the credibility problem with the 
definition and development of a flexible, extensible methodology for creating software ar¬ 
chitecture-based product models supporting the development of authoritative and consistent 
M&S representations in very large-scale software-intensive M&S systems. 

[RGHOO] noted that researchers study fidelity and conceptual models separately and 
independently, although they share many similarities and have a common source of diffi¬ 
culties with a lack of formal, fundamental simulation framework. [RGHOO] also suggested 
a cooperative research effort to understand the mutual relationship between fidelity and 
conceptual models to resolve the common problems. The software architecture-based 
product line"^^ referent view incorporates the characteristics and attributes of a conceptual 
model domain architecture framework. This approach provides a method for the evolution 
of a domain conceptual model referent from an existing heterogeneous Department M&S 
hierarchy, and establishing a quantifiable methodology for establishing M&S credibility 
not accomplished by previous methods. Key elements include: 

• Development of a layered M&S software architecture supporting architecture-based 
systems for improved interoperability, involving reverse-engineering, reuse, and re¬ 
engineering techniques, 

• Identification of a method to evolve an existing heterogeneous M&S domain hierar¬ 
chy into a software architecture-based product line model, 

• Contributing to the development of scalable solutions for building credible M&S 
software solutions, 

• Supporting the development of an alternative M&S framework supporting verifica¬ 
tion and validation employing Architecture Readiness Levels, 


46 

The Department’s Selected Acquisition Report (SAR) summary as of June 2002 identified the cost estimate for the JSIMS program at 

$1.29 billion. 

47 

A product line is a group of products sharing a common, managed set of features that satisfy specific needs of selected market or 
mission [BFG+00]. 
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• Contributing to the development of data quality measures, improved M&S meta¬ 
data'^^, and the body of knowledge to standardize data in support of eommon, eon- 
sistent authoritative representations; 

• Adding to the Department’s Modeling and Simulation Body of Knowledge. 

G. SIGNIFICANCE OF DISSERTATIONS CONTRIBUTIONS 

There are signifieant Department M&S issues to resolve, ineluding heterogeneous 
system representation anomalies. The Department has yet to aehieve the full potential of 
M&S teehnology due to a laek of eredibility in the current M&S practices including con¬ 
ceptual models, the VV&A process, and model quality attributes such as fidelity [YPEOO]. 
Simulation interoperability has also posed significant challenges. All previous M&S inter¬ 
face protocols^'^ and supporting distributed interoperability (e.g., ALSP, DIS, HLA) en¬ 
countered heterogeneous (e.g., substantive interoperability) anomalies and failed to resolve 
the problem. 

Interim solutions such as translators [You02c] may provide near-term relief, how¬ 
ever, they introduce other maintenance, cost, performance, and accuracy risks. In addition, 
integrating translators into simulation federations presents new challenges including exces¬ 
sive latency, or incorrect translations due to syntactical or semantic errors. [TH03] con¬ 
tends that improving several different aspects such as standards, architectures, components, 
data models, and processes may improve interoperability discussed in this research. 

The strategy to replace many of the large-scale legacy M&S with new, better, soft¬ 
ware engineered M&S solutions such as the Joint Simulation System (JSIMS), Joint War¬ 
fare Simulation (JWARS) [BBG+99d], and Joint Model and Simulation System (JMASS), 
discussed in Chapter VII, have not achieved the desired results. All three major Depart¬ 
ment simulation software development efforts to date exceeded initial cost estimates, failed 
to meet schedule, and produced initial results well short of the design specifications. The 
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Data quality is the correctness, timeliness, accuracy, completeness, relevance and accessibility that make data appropriate for use. 
Quality statements are required for source, accuracy (positional and attribute), currency, logical consistency, completeness (feature and 
attribute), clipping indicator, security classification, and releasibility [DoD98]. 
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Metadata is data that describes data and my include definitions, classification, accuracy, data type, precision, and source [DoD95]. 
Protocols are a set of rules and formats, semantic and syntactic, which determine the communication behavior of simulation applica¬ 
tions [DoD95]. 
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introduction of a software architecture-based product line model into the Department’s 
M&S domain has the potential to support: 

• A Department M&S Software Product Line Family Strategy including the devel¬ 
opment of an integrated architectural description^* containing architecture views, 
components, and data models, supporting improved software development of simu¬ 
lations with authoritative representations, 

• An extension to the Department’s architectural framework to support a simulation 
software architecture-based product line model, 

• The development of a software architecture-based product-line simulation metadata 
registries, 

• Additions to the Department’s M&S Body of Knowledge (MSBOK). 

• The evolution and maturation of product line practices in the Department support¬ 
ing improved M&S software engineering processes. 

• Follow-on efforts to develop a composable M&S capability in the Department. 

H. DISSERTATION VISION 

A critical early decision in this dissertation effort involved whether to follow a 
broad or narrow dissertation vision. The objectives of the NPS distance learning Ph.D. pro- 
gram [NPS02] and the scope of the literature search cited in Chapter II established the 
framework for our decision process. There were pros and cons for each approach. A nar¬ 
row dissertation vision supported a research area concerning some specific aspect of the 
Department’s simulation software practice affecting simulation credibility. The narrow 
vision study approach potentially provides more focus, clarity, and precision. However, 
there are many past and present research efforts employing the narrow vision study-scope 
efforts cited in Chapter II. Conversely, there is a dearth of broad vision research efforts 
“related to the development and evolution of complex [Department simulation] software 
systems” [NPS02], indicating a need for a broad vision or strategic level dissertation vision. 

We chose a broad dissertation vision. Two historical precedents reinforce this deci¬ 
sion, the first being the concept of a “silver bullet” solution to the software industry’s woes, 
which languished on for over twenty years before private sector and government executive 


Every system has an architecture, and an architecture can be recorded by an architectural description (AD). IEEE Std 1471-2000 
distinguishes the architecture of a system, which is conceptual, from particular descriptions of that architecture, which are artifacts or 
concrete products [lEEOOb]. 
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The Ph.D. program in Software Engineering is specifically designed for DoD software practitioners who want to acquire the skill and 
knowledge to perform state-of-the art research on issues related to the development of large complex software systems, and to intelli¬ 
gently manage the research of other software practitioners [NPS02] 
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management realized the software quality problem was multi-dimensional. The laek of 
eredibility in Department M&S software is also multi-dimensional. 

The seeond historical precedent involves the industry’s learning curve over the past 
fifteen years from the less-than-successful “small-grain” reuse programs through today’s 
“large-grain” reuse initiatives, including software product lines, a central focus of this re¬ 
search. A broad dissertation vision requires an understanding of the primary and contribut¬ 
ing factors eroding credibility in Department simulations including organizational issues, 
quality models, software engineering, life-cycle models and processes, software architec¬ 
ture, interoperability, and information assurance, to name a few. 

Another factor contributing to the decision to pursue a broad dissertation vision is a 
personal objective to add to the software engineering and software architecture body of 
knowledge to develop the needed skill sets for the future Department Domain and Enter¬ 
prise Software Engineers and Architects. It is an interesting note that many models in dif¬ 
ferent domains have five levels of maturity: Maslow’s Hierarchy of Needs, the Capability 
Maturity Model family, the Cognitive Hierarchy, and the Information Superiority Correla¬ 
tion Model [Gre97]. An additional model, the Eevels of Information Systems Interopera¬ 
bility (EISI) [TEW+00] model also has five levels identifying the levels of interoperability 
maturity (e.g., isolated, connected, functional, domain, enterprise). 

A basic premise of each of these models is that the next higher layer builds on con¬ 
tributions of the previous layer(s). This suggests that if the Department requires future 
Domain and Enterprise Eevel Software Engineers and Architects, a maturation process and 
a multi-dimensional view and understanding of the contributing knowledge bases (e.g. 
software engineering or simulation body of knowledge) is in order. 

The last factor supporting this view is time. The Department’s Transformation 
process has substance, and the pace quickens daily in all areas of the Department to discard 
non-productive practices or systems, better integrate warfighting capabilities, introduce ef¬ 
fective and efficient business practices, reduce major headquarter overhead, and better allo¬ 
cate the Department’s human resources. Credible and timely modeling and simulation ca¬ 
pabilities must quickly evolve to support these many initiatives or risk becoming irrelevant. 
Three recent case studies reinforce this assessment: 
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• The Department completely revamped and revised the non-responsive acquisition 
procedures three times in approximately one year: 5 April 2002 [ASC02b, ASC02c, 
DoD02b, DoD02f], 30 October 2002 [Wol02a, Wol02b, DoD02g, DoD02h], and 12 
May 2003 [DoD03b, DoD03c]. This pace of chance is unheard of in the Depart¬ 
ment’s acquisition bureaucracy. Periodic Department-level studies, Defense Sci¬ 
ence Boards, and Blue Ribbon panels studied the systemic acquisition problems 
many times over the past thirty years and generally provided similar recommenda¬ 
tions each time. For many reasons the Department never enacted most of the far- 
reaching recommendations. 

• The recent war in Iraq reflected this new Transformation thought process in many 
ways including vastly improved sensor-to-shooter decision cycle times, improved 
management and visibility of logistics, and a much closer integration of all Service 
warfighting efforts enabled by information technology [OBB+03}. 

• A recent Department manpower transformation initiative proposes radical changes 
in both the military and civilian management practices of the Department. The pro¬ 
posed civilian management changes affect bureaucratic Civil Service rules, an area 
historically immune to change. [Hay03a]. 

Time is a critical factor. The Nation’s strategic decision cycle time against foreign 
and domestic threats continues to decrease, while new challenges appear every day to test 
all aspects of national power, including the economic, military, political, and psychological 
pillars. A broad vision, strategic view of simulation credibility supports the Department’s 
transformation initiatives, focusing on M&S as a critically needed corporate asset which 
has yet to achieve its full potential. The “silver-bullet” solution languished nearly twenty 
years until discredited as a flawed concept. The challenge involves improving simulation 
credibility supporting confidence in national defense decisions, while concurrently reduc¬ 
ing the development time and complexity for new portable and distributed simulations to 
meet unknown future exigencies. The problem definition suggests a need for broad re¬ 
search approach and evaluation of all germane factors. 

I. DISSERTATION ORGANIZATION 

We adopted a dual-track research methodology, conducting an exhaustive secon¬ 
dary source search to determine the scope of the problem, complemented by primary 
source research of simulation model representations in a specific Department domain. 
Chapter II provides a literature survey on the current research areas relevant to Department 
simulation credibility. 
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The literature survey clearly reveals two major eras affecting Department simula¬ 
tion credibility: (1) the Growth Era before 1994, and (2) the Managed Era, or post-1994 
period, when the Department began to plan, develop policy and procedures, budget, and 
corporately manage the M&S portfolio developed during the Growth Era and plan for new 
joint-simulations to replace many of the legacy Growth Era M&S. This chapter also intro¬ 
duces the Department’s M&S management framework for establishing credible simula¬ 
tions, including classes of simulation, simulation hierarchies, initiatives to establish distrib¬ 
uted simulation capabilities, the expanding need of credibility by the major stakeholders 
influencing the Department’s M&S management priorities: acquisition, training, opera¬ 
tions, analysis, and experimental domains. 

Research indicates that M&S credibility is a multi-dimensioned problem. Chapter 
III through Chapter VII review different dimensions of the Department’s M&S credibility 
challenge. A major objective of this work is to provide a contribution to the next period of 
Department M&S development—the Software Architecture-based Composable era. 

Chapter III introduces M&S credibility issues and identifies concerns that decision¬ 
makers have with Department simulation results, followed by an overview of the Depart¬ 
ments VV&A processes, and several factors contributing to the implementation of, or 
shortfalls of the VV&A processes in the Department. Two major periods, the Growth and 
the Managed Eras characterize the efforts to improve credibility in the Department’s simu¬ 
lation results. 

Chapter IV reviews several heterogeneous challenges to Department M&S credibil¬ 
ity. Improving the credibility of large-scale, legacy Department M&S requires an under¬ 
standing of the underlying systemic causes for heterogeneous system representation 
anomalies, especially in federation interoperability. These diverse challenges include syn¬ 
tactic and semantic heterogeneity, data complexity, interoperability, technical and substan¬ 
tive interoperability, fidelity, and multiresolution modeling. 

Chapter V provides the Department’s current architectural framework for resolving 
the underlying systemic causes generating the pre-conditions for heterogeneous system rep¬ 
resentation anomalies, especially in federation interoperability. The Department’s initia¬ 
tives to improve the credibility of large-scale, legacy Department M&S supporting distrib- 
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uted interoperability inelude the “as-is” architeeture and the evolving “to-be” arehiteeture: 
the Joint Teehnieal Architeeture, and the Common Technical Framework. The Common 
Technical Framework components include the High-Level Architecture, conceptual model 
requirements, and data standards supporting credible simulation development. 

Chapter VI addresses software and simulation quality attributes affecting heteroge¬ 
neous system representation anomalies and credibility in Department M&S, especially in 
federation interoperability. The chapter reviews the internal and external attributes of the 
ISO 9126-1 quality model [ISOOl], testing for quality attributes, and M&S quality attrib¬ 
utes addressed in much of the current literature, including an overview of aggregation and 
disaggregation. The chapter concludes with a discussion on M&S quality approaches. 

Chapter VII reviews three selected areas influencing credibility of large-scale, leg¬ 
acy simulations in the Department including process improvement, the status of new major 
simulation software engineering initiatives (e.g., JWARS, JSIMS, JMASS) and the grow¬ 
ing software engineering body of knowledge. The chapter also includes a discussion of the 
challenges generated by Department contract management oversight for achieving quality 
software and reducing heterogeneous system representation anomalies. Department institu¬ 
tional factors affecting M&S credibility, and recent Department software engineering edu¬ 
cation initiatives. In addition, research reveals a growing awareness that the Department 
needs qualified software engineers who understand the foundations of the software¬ 
intensive systems upon which we basing our future security, economic prosperity, and mili¬ 
tary preparedness, including credible M&S. 

Chapter VIII discusses traditional product lines, and introduces software product 
lines and software architectures. The chapter also reviews core asset development, reverse 
engineering to develop core assets, reengineering core assets, component development sup¬ 
porting a product line methodology, product line practice areas, and an overview of 
evolving software architecture theory potentially applicable to improved M&S credibility 
and reducing heterogeneous system representation anomalies. 

Chapter IX provides an analysis of heterogeneous system representation anomalies 
in five selected M&S in a component of the Nation’s ballistic missile defense domain, the 
Missile Defense Agency. The analysis reviewed the following areas affecting credibility of 
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the five systems under study: organizational ehange impaets, the National Seeurity-based 
need for eredible M&S, risk identification, a domain M&S overview to provide a context 
for the five systems under study, an overview of the five systems under study, and an 
analysis of selected model representations in the five systems under study. Specific analy¬ 
sis areas affecting heterogeneous system representation anomalies and credibility included 
VV&A, fidelity, M&S development demographics, and interoperability. 

Chapter X introduces the detailed research design for the architecture-based product 
line model including the method, specification, design, and implementation of a software 
architecture-based development methodology for a product line referent with multiple 
views and viewpoints. The chapter also introduces the M&S architecture-based product 
line model as an abstract software architecture-based foundation, supported by Architecture 
Readiness Levels (ARL), for resolving heterogeneous system representation anomalies, and 
improving credibility in federation interoperability. 

Chapter XI presents the major conclusions of this research, suggests areas for future 
research, and provides the significance of a software architecture-based product line model 
supporting M&S credibility. We identify possible future research opportunities for the 
software architecture-based product line model methodology not addressed within the 
scope of this research (e.g., composability, managing product line variability). 
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II, LITERATURE SEARCH AND DOMAIN M&S BACKGROUND 


A. INTRODUCTION 

Chapter II provides a literature survey on the current research areas relevant to De¬ 
partment simulation credibility. The literature survey clearly reveals two major eras affect¬ 
ing Department simulation credibility: (1) the Growth Era before 1994, and (2) the Man¬ 
aged Era, or post-1994 period. 

This chapter also introduces the Department’s M&S management framework for es¬ 
tablishing credible simulations, including classes of simulation, simulation hierarchies, ini¬ 
tiatives to establish distributed simulation capabilities, the expanding need of credibility by 
the major stakeholders influencing the Department’s M&S management priorities: acquisi¬ 
tion, training, operations, analysis, and experimental domains. Research indicated that 
M&S credibility was a multi-dimensioned problem. Chapter III through Chapter VII re¬ 
view different dimensions of the Department’s M&S credibility challenge. 

B, LITERATURE SEARCH 

A significant body of literature evolved since the mid-1970s focused on the need for 
major systemic improvements in M&S software credibility as a basis for supporting higher 
levels of confidence of simulation results by Department- and National-level decision¬ 
makers. The taxonomy shown in Eigure 2-1 represents a synthesis of the major research 
areas for models, simulations, verification, validation, and accreditation affecting the credi¬ 
bility of simulations and Department- or National- level decision-makers confidence in the 
Department’s simulation or federation results. 

Unfortunately, there is no “silver bullet” for improving simulation software credi¬ 
bility. Research indicates that improving the credibility of the Department’s computer- 
based M&S software supporting enhanced user confidence involves the following multiple 
interrelated domains, competencies, disciplines, and bodies of knowledge illustrated in 


Eigure 2-1 and discussed in the following chapters: 

• Department of Defense M&S Credibility Measures-Chapter III 

• Heterogeneous System Representation Anomalies-Chapter IV 

• The Architectural Eramework-Chapter V 
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• Simulation Software Quality Attributes-Chapter VI 

• Proeess Improvement and Software Engineering-Chapter VII 

• Software Product Lines and Architecture-Chapter VIII 

• Domain Analysis-Chapter IX 



Figure 2-1. Major Research Areas Relevant to this Work 


M&S credibility has also taken on additional importance and emphasis as the 
Department employs large-scale, legacy M&S supporting Department missions and 
national security initiatives. The literature survey found the use of M&S software 
pervasive in many domains, expanding rapidly into many new areas, and experiencing a 
fast-growing demand for even more credible applications and uses: 

• United States Homeland security and national security strategy lBus02a, Bus02b, 
Bus02c, Bus02d]; 

• Department policy, memoranda, and strategy documents [Kri88, Car88, DOT89, 
Kri89, Pai97, Gan98, Gan99, DoDOOb, GMWOOa, GMWOOb, MonOOa, MonOOb, 
SheOOa, HRA+01, WolOl, RumOl, Rum02a, Rum02b, Ald02a, Ald02b, Ald02c, 
Ald02d, DoD02d, DoDOSa, DoDOSb, DODOSc]; 
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• United States Government Accounting Office (GAO) Reports [PAD78, PAD79a, 
PAD79b, PAD80, Mye82a, Mye82b, She82, Con86a, Con86b, Che87, Bre88, 
Bro88, Bro91, Dav91, HSW+92, Con93, Geb93, Geb94a, Geb94b, Geb95, Dav96, 
ZHF98, Bro99, HDK+00, HEE+00, McCOO, REH+00, BCP+01]; 

• Defense and Army Science Board studies [WHM+79, JBC+88, BTE+93, HW96, 
GKB+97, EWE+98, HWB+98, MCC+98, BBB+99, EEE+99, EGW+99, GBB+99, 
GWB+99, HAC+99, EED+99, MSI+99, HNC+00, MorOOb, BCC+Ola, EOP+01, 
VEH+01]; 


• Private and Government sector studies, analyses, and reports [Qua64, SCC+88, 
DOT89, DB91, LA92, Dow92, DH92a, DH92b, ABD+93, CSC94a, CSC94b, 
Coh94, SG94, Bec94, EBG+94, Par94, PBA+94, PMR94, JAS95, MCU+95, 
HBM96, PHP+96, Por96, WSM+96, CCK+96, JAS96, JAS97a, JAS97b, JAS97c, 
Eie97, Gau98, NMY+98, SG99, BCE+99, DOTOO, SBAOO, BEOO, COHOO, 
MWD+00, DMSOOb, RGHOO, CraOla, CraOlb, DDHOl, DOTOla, DOTOlb, 
HarOla, HarOlb, HCG+Ola, HCG+Olb, NelOl, SBAOla, SBAOlb, RTI02]; and 


• Department Inspector General reports [RSV+93, BSV+97, GGEr+97, AMD98, 
GUS+OO], 

M&S software credibility areas that affect decision-makers’ confidence in simula¬ 
tion results and authoritative model representations: 


• Software engineering and development processes based on best practices [You89, 
BL91, You93, Coh94, Dem95a, Som95, HCP96, Hoe96, Wie96b, Pre97, HBE97, 
Hin97, CS97, BW97, Rub97, REE+98, Roa98, Woo98, HHK+99, BasOO, RSE+00, 
BDA+00, HHPOO, HNC+00, REH+00, BCC+Olb, KMOla, RakOl, CHS+02, 
MR02, Ei02, EQZ02]; 

• Process improvement and reengineering [Sch91, HC93, Hal95, HamOlb, 
PWC+95, EC99, MCR+00, BBE+01]; 

• Modeling and simulation improvement [BKR95, Ham96, HNP97, Hug97a, CR98, 
Koo99, HamOla, LKOO, BCB02, Bee02]; and 

• Emerging bodies of knowledge (BOK) in related knowledge areas such as soft¬ 
ware quality measurement [HM96, Sch02a], project management [Era94, Dun96], 
and modeling and M&S BOK [ACP+99], the Department’s joint technical architec¬ 
tures [JTA02a, JTA02b], sound knowledge engineering practices for conceptual 
model development [Pac99a, PacOla] and verification and validation [IEE86, 
IEE98b] of the conceptual model, data, and simulations [Pac02a, Bor02] conducted 
during the entire M&S life-cycle. 
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A top-level literature survey was also eonducted for: 

• Studies in the Department’s Missile Defense M&S domain [Bra96, Li99, ISE99, 
WelOO, BraOOa, BraOOb], 

• Performance recommendations supporting improved information technology in¬ 
vestments within the Department and federal government: [SBD-i-95, BPC+96, 
MLW+96, AVL+97, Hin97, Pai97, Dod98, PKM+98, Bro99, MecOO, RodOO, 
RSF+00, CLP+01, CHS+02, FNP+02, BBF02]. 

• The scope and magnitude of simulation software credibility. The problem with 
simulation eredibility appears to be a systemic issue extending well beyond the De¬ 
partment’s purview, and found in different domains and diverse, non-Department 
systems such as economic models [PosOO, KBA-l-02], transportation [AndOO], en¬ 
ergy [Sta74], supply [PSA79], environmental protection [DMF-l-95], retirement 
forecasting [Dod97, Che86a, Che86b], supercomputing [Bro91], air pollution 
[GMW89, DMP-l-97], credit subsidies [Cal97], highway management [SchOla], nu¬ 
clear weapons [CFM99, HFC91], and manufacturing [AROO]. 

Finally, the Department M&S software programs must support and comply with 
applicable Federal Government and Department requirements, software development proc¬ 
esses and product security measures including: 


• Information operations / assurance and critical infrastructure protection re¬ 
quirements [FFF+97, FX99, ABBOO, BusOl, FOP+01, GPH+01, WFPOl, AFF02], 

• Interoperability standards [HKF96, HMD99, MCF-i-99, HMOO, BFB+Ol, 
BFR+02], and 

• Legal requirements for information technology and performance measurements 
[CC96, ATFOl, CosOl, GlaOl, Man02, AS02a, AS02b]. 

C. THE DEPARTMENT’S MODEL AND SIMULATION FRAMEWORK 

1. Background 

National- and Department-level decision-makers need high-quality computer-based 
simulation models and modeling frameworks, supported by valid qualified and quantifiable 
levels of process and product credibility. The quest for confidence covers an ever- 
expanding range of M&S ranging from high-fidelity engineering models to highly aggre¬ 
gated campaign-level M&S. The potential spectrum of Department uses for M&S grows 
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continuously when one considers that “everything is a simulation exeept com¬ 
bat,’’[BTE+93]. The terms model^^ and simulation^"^ have several definitions, including 
Law and Kelton [LKOO]^^, the IEEE [IEE90]^^, and the Department [DoD94]. In this re- 
seareh we use the Department’s definitions: a model “is a physieal, mathematical, or oth¬ 
erwise logical representation of a system, entity^’, phenomenon, or process” [DoD94], 
while a simulation is “a method for implementing a model over time” [DoD94]. 

John von Neumann first proposed the concept of modern digital simulation as the 
applieation of gathering repetitive, statistical data on modeled phenomena, via the Monte 
Carlo method [Rot83]. [Tho97a] provides an historieal account of computer-based simula¬ 
tion from the 1940s wartime combat operations researeh role and focus on improving the 
use of weapons, through the 1960s use of M&S for systems analysis. Erom 1940 to 1997, 
there have been four major eras in the evolution of software during the computer eras af- 
feeting the development of Department M&S software [Pre97]: 

• The early years, characterized by custom software, batch operations and limited 
software distribution, 

• The seeond era, from the mid-1960s to the late 1970s, experienced software prod¬ 
ucts developed by the private seetor, multi-user computer systems, on-line data¬ 
bases, and the first real-time systems, and the need for software maintenance grew. 


A model is a mathematical representation of a system, where a system is any sort of process or structure. A process model normally 
operates sequentially over time, but may use other dimensions. Sequential processes in time are modeled as an independent variable over 
a linearly ordered set using real numbers (continuous time) or integers (discrete time). A model has two types of system parameters, or 
variables characterized by range (the set from which it assumes a value) and type (random or deterministic). A deterministic variable 
assumes a unique from the range at each point in time, while a random variable is described statistically probability distribution over the 
range. System variables may be for input or output, and the resulting model may be static or dynamic. In a static model, the outputs at 
any time are only a function of the inputs at that time, possess no memory and has no recollection of previous events, whereas the outputs 
of a dynamic system, may depend upon past values, as well as present inputs, thus the system remembers certain information about pre¬ 
vious inputs. The remembered information is called the state of the system. A structural model usually involves the representation of 
relations e.g., airline flight route [Mar83a] 
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A simulation is the representation of certain features of the behavior of a physical or abstract system by the behavior of another sys¬ 
tem. Simulations employ computerized models of certain significant features of some physical or logical system. The object of the simu¬ 
lation process is to provide an experimental model for the accumulation of data on the target system, and is comprised of the following 
steps: experiment definition, modeling, computer implementation, validation, and data gathering [Rot83]. 

The technique to imitate or simulate the real-world operations of facilities, processes, or systems supporting scientific study; which 
require a set of assumptions about how it works. The assumptions, usually mathematical or logical relationships constitute a model, used 
to gain an understanding of how the corresponding study behaves. A simulation employs a computer to evaluate a model numerically, 
collect data to estimate the desired true characteristics of the model [LKOO]. 

The IEEE defined a simulation as “(1) ^ model that behaves or operates like a given system when given a set of controlled inputs; (2) 

the process of developing or using a a model as in (1) [IEE90]. 
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An entity is a distinguishable person, place, unit, thing, or concept about which information is kept [DoD98]. 
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• The third era, from the mid-1970s lasted over a decade and ushered in a period of 
low-cost computation power for consumer use, saw the development of distributed 
systems, and witnessed the initial efforts into artificial intelligence development, 

• While the fourth era, from the mid-1990s to the present, included mobile and dis¬ 
tributed systems, the Internet, network/parallel computing, expert systems, neural 
networks, powerful desktop personal computers, object-oriented technologies, and 
the software to integrate legacy systems [Pre97]. 

During these four periods the M&S community experienced the development of new com¬ 
puter-based models and simulations, a growing concern with credibility, emerging trends of 
repeated use of models for different applications and the development of families of models 
or hierarchies. 

2, Classes of Simulations 

The current Department M&S framework has three M&S classes: live, virtual, and 
constructive, supporting five major customer domains: training, operations, analysis, ex¬ 
perimentation and acquisition [DoD02d]. Live simulation refers to real people operating 
real systems including field training exercises involving troops and actual equipment, 
which allow soldiers to use organizational equipment under actual environmental condi¬ 
tions, simulating combat. The live simulation also provides ground test data on actual 
hardware and software performance in an operational or development environment. Live 
test data supports simulation validation (e.g., calibration) using the model-test-model con¬ 
cept employed in the Department’s acquisition process. 

Virtual M&S involves real people operating simulated systems, digital representa¬ 
tions of environments, organizations, systems, other entities, and processes. Virtual M&S 
put the human in the loop in a central role by exercising motor control skills, decision 
skills, or communication skills. Current state-of-the-art virtual M&S bring the system or 
subsystem and its operators together in a synthetic or simulated environment. Virtual M&S 
run in real time immersing players in synthetic environments or simulators. Virtual M&S 
may include actual hardware, stimulated by the output of computer simulations. 
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Constructive M&S involves simulated people operating simulated systems. Con- 
struetive simulations employ digital representations of environments, organizations, sys¬ 
tems, other entities, and proeesses, operated with human players, or without human players. 
Human player interactions are limited to eontrolling units, and generally referred to as 
wargames. Human-in-the-loop (HIL) simulations provide the stress and deeision-making 
assoeiated with live simulations, and permit the introduetion of multiple types of platforms 
for evaluation of aetual interaetion and interoperability. 

Categories of eonstruetive M&S eonsist of eomputer models, wargames, and ana- 
lytieal tools used aeross a range of aetivities. These aetivities inelude detailed engineering 
design [Law02] and eost [KanOl] or subsystem and system performanee ealeulations to 
support development of technical specifications. Higher-level eonstruetive M&S provides 
information on the outeomes of battles or major eampaigns involving joint or eombined 
forees; identify mission needs; and support operational effectiveness analyses measures of 
effeetiveness (MOE), loss exehange ratios (LER), foree exehange ratios (EER), and meas¬ 
ures of performanee (MOP). Stoehastie^^ or deterministie^^ simulations approximate the 
real world. Other uses of eonstruetive M&S inelude developing life cyele eost estimates, 
supportability analyses, and foree management analysis of alternatives. 

3. Simulation Model Hierarchies 

The eoneept of a hierarehy of simulation models evolved in Europe and the United 
States during the 1970s with the objective of managing M&S eomplexity^®. However, the 
various hierarehy models laeked integration and usually eonsisted of a high-level model 
and several lower level M&S. [Ham96, HNP97] suggest that hierarehieal modeling sup¬ 
ports abstraetion^^ A variable resolution M&S eoneept for integrating heterogeneous 
M&S into the hierarehy has existed sinee the early 1980s [DH92a, DH92b]. 


The theory of stochastic processes deals with events that develop in time or space and which cannot be described precisely except in 

terms of probability theory [Mar83b]. 
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A deterministic variable assumes a unique value from the range of set values at each point in time [Mar83a]. 

Complexity refers to time and space resources needed to simulate a model, and is a dimension independent from resolution, although 
they are often related in a monotonic manner [Zei92b] 

Abstraction isolates important aspects of the problem and suppresses the unimportant aspects [HNP97]. 
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[HHPOO] suggests the literature includes many other different M&S taxonomies. 
For example, [Zei92a] addressed model representations in modular hierarchies, while 
[Sch92] detailed model hierarchies of different resolutions. Providing other viewpoints, 
[HBF92] focused on maintaining process independence between different levels of resolu¬ 
tion; while [BD02] identified hierarchical model framework to support improved design 
quality of object-oriented systems; whereas [Rei92] adopted a different five-level capabili- 
ties assessment viewpoint . Lacking an architectural foundation, many of these hierarchy 
models depended on broad categorical definitions, heuristics, subject matter experts, and 
general qualifications to match an existing M&S to the model framework. 

[Hug97a, Hug97b] proposes there is not a universally accepted way to categorize 
M&S, since M&S range over a vast spectrum of fidelity, precision, and resolution from de¬ 
tailed engineering representations to highly aggregated representations of force-on-force 
engagements, often involving multiple dimensions of structural, functional, temporal, and 
qualitative abstraction, which are not complete or necessarily independent of each other. 
[Ham96, HNP97] cite the close relationships between the concepts of aggregation and de¬ 
composition with the levels of abstraction concept. [Ham96, HNP97] also note that ab¬ 
straction (e.g., ignoring behavior or minor differences) indicates a simplification strategy, 
whereas abstraction by lumping together indicates the use of an aggregation method. 
[Ham96, HNP97] further suggest the use of an open architecture as a means of developing 
simulations with the flexibility to support multiple levels of aggregation. 

With the Department’s growing need to address human behavior, [MTOl] proposed 
a hierarchical model composed of virtual individuals, groups, and crowds. [Sta97] pro¬ 
vides a different perspective, suggesting that a hierarchy of linked measures of merit— in¬ 
cluding measures of performance, measures of functional performance, and measures of 
force effectiveness— may improve acquisition activities. Finally, [PMR94] introduced a 
Notional Hierarchy of Technologies^^ with five categories for assigning M&S enabling 


The capabilities assessment levels built on weapons system level analysis, and added a supporting analyses level, program alternative 
analyses, mission analyses, and at the top of the pyramid, campaign analysis [Rei92]. 

The enabling technologies included standards and protocols at he lowest level, and added layers for underlying collaborative tech¬ 
nologies, utilities and infrastructure, applications, and an integrated acquisition environment at the top [PMR94]. 
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technologies: standards and protocols, collaborative technologies, utilities and infrastruc¬ 
ture, applications, and at the top, an integrated acquisition environment. 

The Department’s simulation hierarchy includes four categories of M&S for com¬ 
municating the scope or level of model detail: engineering, engagement, mission and cam¬ 
paign, [PMR94, CS97], generally with more abstraction moving up the simulation hierar¬ 
chy and more refinement, or level of detail moving down the simulation hierarchy. The 
M&S characterized in the hierarchy represents increasing aggregation from the engineering 
models at the base of the hierarchy to the highly aggregated representations supporting a 
theater level view [HM95]. Not every model or simulation fits perfectly into one of the 
classifications, and M&S fidelity quality, addressed later, can vary at any point in the M&S 
hierarchy. 

The engineering M&S include very detailed simulations, concentrating on individ¬ 
ual components and their interaction, with application in design analysis, risk mitigation in 
component performance and tradeoff, requirements specification, and performance analy¬ 
sis. Engagement M&S usually depict friendly versus enemy engagements, and generally 
provide some type of system effectiveness outcome supporting analysis of alternatives 
(AoA), requirements evaluation, system effectiveness, tactics, techniques, and procedures 
(TTP) development and assessments, tradeoff analysis, and test support. Virtual prototypes 
usually run as a simulation at the engagement level and above. 

Mission/Battle level M&S depict multi-platform, multi-task force packages and 
usually provide some type of mission effectiveness outcome. Mission/Battle M&S support 
AoA and requirements evaluation, deployment analysis, weapons integration, TTP devel¬ 
opment and assessments, training, and wargaming. At the apex of the M&S hierarchy sit 
the highly aggregated Theater/Campaign models. These M&S also support AoA, require¬ 
ments evaluation, TTP development and assessment, war gaming, and battle-staff training. 
Figure 2-2 is an annotated illustration of the current Department M&S hierarchy [PMR94]. 
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Figure 2-2. Department M&S Hierarehy (After [PMR97]) 


4. Distributed Simulations 

The Department supports three arehitectural implementations for distributed M&S: 
the Aggregate-Level Simulation Protoeol (ALSP), the Distributed Interactive Simulations 
(DIS)^"^ application protocols [IEE95, IEE98a], and the most recently approved method, the 
HEA. Eor distributed simulations [Ham96, HNP97], an individual simulation called a fed¬ 
erate interoperates with other selected federates through a common technical framework, 
such as the HEA, forming federations. 

The IEEE supports the HEA rules, IEEE 1516/Dl [IEE98e], HEA object model 
template specification [lEEOOc], HEA federate interface specification [lEEOOd], and HEA 
object-model template [lEEOOe]. The HEA object-model-template specification [lEEOOe] 
defines the format and syntax of HEA object models, but not the content. With the evolu¬ 
tion of the HEA beyond the Department, a growing demand for interoperability, and ap- 
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DIS is a synthetic environment within which humans may interact through simulations at multiple sites networked sing compliant 
architecture, modeling, protocols, standards, and databases [DoD95]. 
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proval of the HLA standard by the IEEE, the definitions employed in this research will re¬ 
main consistent with approved IEEE definitions (e.g., accuracy^^, resolution^^) whenever 
possible. Legacy Department definitions included in the research provide context for deci¬ 
sions, issues, and possible ambiguities faced by Department simulation developers. Inter¬ 
estingly enough, the term fidelity, previously defined by the Department [DoD98], and used 
liberally in the M&S literature, is not included in the IEEE’s standard glossary of software 
engineering terms [IEE90], or the HLA object model template specification [lEEOOe]. 

5. The Growing Need for Credible Models and Simulations 

As M&S technology developed in the Department, several functional areas, includ¬ 
ing the acquisition, training, operations, analysis, and experimental functional domains in¬ 
corporated simulation technology into their infrastructure with the goal of enhancing per¬ 
formance, reducing overall costs, and attaining improved schedule milestones. In all of 
these functional domains, M&S is now a key component of the domain toolbox, although 
the past performance of simulation technology has not always met expectations. 

a. The Department's Acquisition Domain 

The Department’s acquisition domain, including developmental and opera¬ 
tional test and evaluation communities, employ M&S in the acquisition of new operational 
systems [DoD03b, DoD03c]. The Department also incorporated M&S into the acquisition 
process to reduce cost, improve performance, maintain schedule and mitigate risk. The re¬ 
quirement for credible M&S results are incorporated in the Department acquisition direc¬ 
tive [DoD03b] and instructions [DoD03c], which state, 

• The conduct of test and evaluation, integrated with modeling and simula¬ 
tion, shall facilitate learning, assess technology maturity and interoperabil¬ 
ity, facilitate integration into fielded forces, and confirm performance 
against documented capability needs and adversary capabilities as described 
in the system threat assessment [DoD03b]. 


Accuracy according to the IEEE is the measure of the maximum deviation of an attribute or parameter value in the simulation or fed¬ 
eration from reality or some other chosen standard or referent [lEEOOe]. 

The IEEE defines resolution as the smallest resolvable value separating attribute or parameter values that can be discriminated. Reso¬ 
lution may vary with magnitude for certain data types [lEEOOe]. 
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• The T&E [test and evaluation] strategy shall provide information about risk 
and risk mitigation, provide empirical data to validate model [DoD03c] 

• Adequate time and resources shall be planned to support pre-test predictions 
and post-test reconciliation of models and test results, for all major tests. 

[DoD03c]. 

In addition, “The PM shall pan for M&S resources throughout the acquisition life 
cycle. The PM shall identify and fund M&S resources early in the life cycle” 

[DoD03c]. 

In the area of acquisition reform [WHA+OO], the National Performance Re¬ 
view (NPR) established a Department acquisition reform objective of reducing new system 
delivery time by twenty-five percent, subsequently increased by the Department to fifty 
percent [ACP+99], in part by acquisition reform initiatives capitalizing on engineering 
trade-offs and M&S performance modeling to achieve the NPR goals. Simulation software 
also supports the evolving Department transformation^’ objectives [OAH-l-97, DGH-l-98, 
RumOl, Rum02a, Rum02b] to improve the quality, interoperability, effectiveness, cost, 
performance, and schedule metrics of the Department’s business and warfighting systems 
supporting the National Military Strategy [HBM96, Rum02b]. 

In the acquisition arena, transformation includes the replacement of inflexi¬ 
ble requirements-based methods with a capabilities-based methodology [RumOl] supported 
by evolutionary acquisition practices [Ald02d] for major Department acquisitions including 
the Nation’s missile defense program. The success of M&S to support the Department’s 
transformation process hinges in part, on the acceptance and successful implementation of 
new or enhanced M&S methods and techniques such as the Simulation, Test, and Evalua¬ 
tion Process (STEP) [CS97], Simulation-Based Acquisition (SBA) [San97b, San97c, FT98, 
JMS98, San98b, SBB+98, Bro99, GS99, San99, SBA99, SBB+99a, SBB+99b, ATOO, 
SBAOO, BGK+OO, KLM+OO, KMOlb, ZitOl, ECP02], Distributed Product Descriptions 
(DPD) [HHOOb], the Army’s Simulation & Modeling for Acquisition, Requirements, and 
Training (SMART) program [BiaOO, EKHOO, EKHOl], and the development of new col¬ 
laborative environments [San98a, San98b, San99, HolOl] including the Joint Synthetic Test 

Transformation results from the exploitation of new approaches to operational concepts and capabilities, the use of old and new tech¬ 
nologies, and new forms of organization that more effectively anticipate new or still emerging strategic and operational challenges and 
opportunities and that render previous methods of conducting war obsolete or subordinate [RumOl] 
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and Evaluation Battlespace [San98b, Bra98, HolOl], the Advanced Distributed Simulation 
(ADS) [Kec99] capability, and testing system-of-systems interoperability (e.g., the Single 
Integrated Air Picture) [Dah02]. 

The Revolution in Business Affairs, a Department acquisition reform im¬ 
perative, leverages M&S to reduce Total Ownership Cost and improve delivery for new 
systems by employing SBA processes [Eva96, Eav97, San97b, BSB+98, HolOl] among 
other best business practices. The vision for the Department’s SBA initiative is “an acqui¬ 
sition process in which the Department and industry are enabled by robust, collaborative 
use of M&S technology integrated across acquisition phases and programs” [San97b]. The 
objective of SBA is to reduce schedule time and risk in the development and maintenance 
of a product with an improved quality at a lower cost in a collaborative government/private 
sector environment employing an Integrated Product and Process Development methodol¬ 
ogy over the system life cycle. However, Department support for these initiatives requires 
the development of business-case framework to evaluate the investment opportunity and 
the potential return-on-investment [BGK+00]. 

STEP is an iterative model-test-model approach with a goal of improved 
M&S credibility, test community confidence, and a better understanding of the strengths 
and limitations of complex models, M&S and data by decision-makers. The STEP para¬ 
digm of model-simulate-fix-test-iterate [CS97] embodied in the DoD developmental and 
operational test and evaluation (T&E) process has the potential to support improved confi¬ 
dence in tests and evaluation results. 

Department testers [DOT89, Bra98, Kec99, BCE+99, Pau99, Obr99, 
CoyOOa, CoyOOb, DOTOla, MOR02] also seek answers to where M&S may provide credi- 
ble information and under what conditions M&S can replace testing. Hydrocode models 
track shock propagations and materiel distortion to support vulnerability/lethality assess¬ 
ments [TIL+95, MH98] and support Department Eive Eire Test & Evaluations (EET&E). 
The EFT&E testers characterize the results of fragment and kill-vehicle interaction with the 
target, which are subsequently incorporated into lethality M&S. Current research into hy- 


Hydrocodes are physics-based, first principle M&S of hyper velocity solids interaction used in vulnerability and lethality M&S to 
track shock propagation and distortions of multiple materials [IDAOl] 
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drocodes models cited by [IDAOl] indicates current hydrocode model technology is limited 
to the primary effects of a weapon on a system and has major voids in secondary weapons 
effects, suggesting that M&S can complement, but not replace live-fire tests. 

[DOTOlb] cites the quality of Department software testing as an ongoing 
systemic issue, needing major improvements in the area of M&S support for improved test 
and evaluation readiness capabilities. [CoyOOa] supports this assessment and suggests ad¬ 
ditional emphasis in the area of best software test processes to improve the quality of the 
Department’s operational testing program. However, [BCE-l-99] notes that simulating 
complex system interactions, limited by cost, time, personnel, and technology, adversely 
impacts the accuracy and fidelity of the M&S supporting Departmental test programs. 

A Departmental M&S survey [DDHOl, HBD+Ol] sponsored by the Direc¬ 
tor, Operational Test and Evaluation (OT&E) to gain insight and confidence, performed a 
limited review of Department M&S including types, characterization, management, man¬ 
agement activities, cost, and noted for VV&A that much policy guidance has been devel¬ 
oped, but implementation appears minimal. The increasing dependence on M&S noted by 
[DDHOl] added increased emphasis on management activities considered critical to suc¬ 
cess including: 

• M&S Support Plan 

• M&S Staff 

• VV&A Plan / processes 

• M&S reuse 

• Collaborative environment 

• Performance incentives [DDHOl, HBD+Ol]. 

The Eebruary 2001 Department M&S Workshop for Assuring M&S Credi¬ 
bility for Defense Acquisition and T&E: Survivability, Eethality and Mission Effectiveness 
[DOTOla] corroborated the results from [DDHOl, HBD+01]. The Workshop identified 
several systemic, ongoing factors hampering M&S and VV&A programs, and concluded 
that acquisition program managers need more help implementing M&S and VV&A within 
funding constraints [DOTOla]. Eurthermore, [DOTOla] cautioned there is “no definitive or 
reliable link between following M&S and VV&A policy and technical guidance and having 
confidence in M&S credibility for acquisition applications” [DOTOla]. As a result of these 
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systemic issues, [DoDOla] concluded that the lack “of adequate VV&A resulted in uncer¬ 
tain or poor representations of military forces in M&S, thereby causing negative training, 
inaccurate analyses, and lack of trust in M&S” [DoDOla], 

An October 2002 Military Operations Research Society (MORS) M&S 
Workshop [GF02] addressed relevant topics to this study including; the status and health of 
the Department’s M&S policy, the capability versus expectations for using M&S in De¬ 
partmental acquisition, the Department’s VV&A practices, and addressed the state of the 
Department’s M&S credibility since [DDHOl, HBD+01, and DOTOla], The workshop 
members concluded that the Departmental M&S policies were adequate and identified sev¬ 
eral areas needing additional emphasis including risk mitigation, operational assessments, 
and test-prediction and post-test reconciliation [GF02]. 

The workshop members also noted that most of the Department’s M&S ef¬ 
fort appeared in areas requiring the highest fidelity, although current Departmental M&S 
policies or funding levels do not support these efforts. Specific areas of concern included; 
evaluations beyond the bounds of a test, system performance prediction methods, and the 
expanded use of synthetic test environments in lieu of live tests [GF02]. Numerous VV&A 
issues included; 

• The continuing perception that VV&A is too costly, 

• VV&A not adequately integrated in the M&S process, 

• Programs lack independent V&V, 

• VV&A lacks up-front estimation of the effort, causing unnecessary work, 

• VV&A lacks multidisciplinary membership; and 

• The model-test-model and model reuse paradigms had not evolved as anticipated 
[GF02]. 

The workshop findings according to [GF02] also noted the failure of De¬ 
partment M&S users, including decision-makers to understand the problem or question un¬ 
der review, contributing to the unfocused use of M&S, unbounded VV&A, and the unnec¬ 
essary expenditures of resources. [GF02] proposed examining options for program office 
incentives supporting effective M&S use, improved reuse, and development of M&S for 
cross-program use. 
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b. The Department’s Training Domain 

The Department uses M&S aeross a broad speetrum of training and eduea- 
tion from individual platform simulators to Joint Task Force operational- and strategic- 
level exercises employing multiple simulations replicating a theater of war [DoDOSa], 
Success of the Department’s transformation process also depends upon the transformation 
of the current Department training paradigm [OPV+02] as directed by the Defense Plan¬ 
ning Guidance [DPG02] for fiscal years 2003-2007. Achieving the strategic goal of a train¬ 
ing transformation requires a common operational architecture providing interoperability of 
live, virtual and constructive training and mission-rehearsal systems “to build unparalleled 
military capabilities, which are knowledge-superior, adaptable, and lethal” [OPV+02]. 

M&S also has the potential to greatly improve critical Department program 
initiatives involving joint/coalition doctrine development, training, operations, force man¬ 
agement, requirements development, materiel readiness, life cycle/ total ownership costs, 
human behavior, education, mission planning, research and development, modernization, 
analysis, contracting, and joint warfare experimentation, although at this point a scientifi¬ 
cally definitive conclusion on its effectiveness cannot be drawn. 

c. The Department’s Operations Domain 

Department M&S supports deliberate planning, crisis course of action 
analysis, and mission rehearsal. The Department’s draft Mot/e/mg and Simulation Strate¬ 
gic Plan 2003-2012 [DoD03a] provides a challenging vision statement for Department 
M&S efforts citing the need to: “Provide the Power to predict. Prepare, and Respond Rap¬ 
idly to Win Decisively Against Any Threat” [DoD03a]. Software and information support 
the goal of twenty first century military decision superiority identified by [Rum02b] for 
U.S. forces to communicate with each other, share information, and see the same, precise, 
real-time picture of the battlespace, while simultaneously sharing knowledge of friendly 
and enemy locations. 

Furthermore, the Department determined that M&S “has the potential to 
significantly enhance the military capability and readiness of US forces” [DoDOla]. In or- 
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der to achieve these goals, “DoD must also undertake high-fidelity transformation exereises 
and experiments that address the growing ehallenge of maintaining space control or de¬ 
fending against attacks on the U.S. national information infrastructure” [RumOl]. Improv¬ 
ing the Nation’s missile-defense eapability is a major pillar of the National Security Strat¬ 
egy [Bus02d] and Homeland Defense [Bus02c]. Capability-based aequisition [DoDOSb, 
DoDOSc] places additional requirements on future M&S requirement including the need to 
model a potential adversary’s limitations and strengths. 

There is also an inereasing Department demand for by the Department‘s 
warfighting community for credible and interoperable new modeling and simulation pro¬ 
jects supporting the development of the four major Joint Vision 2020 [SheOOa] operational 
eoneepts - dominant maneuver, preeision engagement, focused logistics, and full dimen¬ 
sion protection^® [SheOOa], linked by three critical interdependent faetors; interoperability, 
innovation and deeision superiority. Interoperability, based on open systems arehiteeture, 
is the keystone of future military operations and “the foundation of effective joint, multina¬ 
tional and interagency operations” [SheOOa]. Deeision superiority is “.. .superior informa¬ 
tion eonverted to superior knowledge to achieve.. .better decisions arrived at and imple¬ 
mented faster than an opponent can react” [SheOOa]. The eoneept of military decision su¬ 
periority has roots in the very genesis of warfare. From warfare’s inception the world’s 
military forees continually refined decision superiority to achieve a strategic advantage or 
victory in battle. In the future, M&S will provide real-time decision support during mis¬ 
sion execution [DoDOSa] 

A recent Department initiative, the Command, Control, Computer, Commu¬ 
nication, Intelligence, Surveillance and Reconnaissance (C4ISR) Information Superiority 
Modeling and Simulation Master Plan for improving information and deeision superiority, 
plans for environments “constructed, to the extent possible, from affordable, reusable com- 
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The scope of this effort includes the adversary’s Political, Military, Economic, Social, Infrastructure and Information (PMESII) 

[DoD03a]. 
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Dominant Maneuver is the ability of joint forces to gain positional advantage with decisive speed and overwhelming operational 
tempo in the achievement of assigned tasks. Precision Engagement is the ability of joint forces to locate, surveil, discern, and track 
objectives or targets; select, organize, and use the correct systems; generate desired effects; assess results; and reengage with decisive 
speed and overwhelming operational tempos as required, throughout the lull range of military operations. Focused Logistics is the ability 
to provide the joint force the right personnel, equipment, and supplies in the right place, at the right time, and in the right quantity, across 
the lull range of military operations. Full Dimension Protection is the ability of the joint force to protect its personnel and other assets 
required to decisively execute assigned tasks [SheOOa]. 
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ponents interoperating through an open systems architecture” [DoD02d]. [SH96] proposed 
organizing around information enabling distributed operations with a decentralized deci¬ 
sion-making process, as a means to establish the environment for quality and speed. How¬ 
ever, issues with M&S reuse and interoperability in the Department indicate that the “value 
of past investments in M&S is depreciated because the DoD does not have the capability to 
easily identify algorithms, data sets, models or M&S that can be reused” [DoDOla]. 

The Department requires additional resolution, level of detail and refine¬ 
ment to existing M&S tools, especially for opposing-force representations of entities, 
which “must be significantly increased to permit realistic assessments of C4ISR perform¬ 
ance [and] the degree of fidelity and detail of M&S of C4ISR functions be commensurate 
with this importance” [DoD02d]. The Department also requires increased fidelity and 
M&S with “more accurate representations of all elements of the mission space” [DoDOla], 
Although the Department made progress in the area of developing authoritative representa¬ 
tions, [DoDOla] notes “many combat, combat support, and combat service support systems 
abound, making it difficult to determine which are authoritative or best of breed.” 
[DoDOla]. 


d. The Department's Analytical Domain 

The military operations research community advanced operational decision¬ 
making over the past fifty years [You97 MK98], employing computer-based models and 
software M&S as “tools supporting an analytical process with better decisions being the 
objective” [Hug97b]. [Pre97] suggests that software supports a multitude of uses, includ¬ 
ing M&S, and delivers information, which may be the most critical product of the twenty- 
first century. Analysis supported by M&S occurs in every Department domain. In addi¬ 
tion, the Department uses M&S in analysis or wargames supporting force structure analysis 
and long-range strategic planning, both major Department activities. 
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e. The Departmenfs Experimental Domain 

The Department also uses M&S to explore new eapabilities, portray future 
operational environments and develop new taeties, teehniques, proeedures, doetrine, organ¬ 
izational struetures, proeesses, and systems [DoD03a]. The Revolution in Military Affairs 
initiatives employ large-seale, legaey M&S, future evolutions of legaey M&S software sys¬ 
tems, and new generations of live, virtual and eonstruetive DoD M&S, for experimentation, 
advanced-eoneept exploration, analysis, and requirements elieitation. These initiatives 
support expanded joint, eombined, or multinational federations, enable M&S-based aequi- 
sition, and advanee development of distributed system engineering eollaborative environ¬ 
ments. 

Department M&S users require timely simulation applieations, improved 
analysis eapabilities, and improved operational support with M&S integrated with go-to- 
war systems, ineluding international eollaborative initiatives and eoalition operations 
[DoD03a]. [RumOl] eited M&S in the 2001 DoD Quadrennial Defense Review as critieal 
for future joint, combined and coalition operations and the dynamic military transformation 
process to the point where “U.S forces rely heavily on war games and M&S to support this 
program of field exercises and experiments” [RumOl]. 

Millennium Challenge 2002 (MC02), the most elaborate war game ever 
conducted by the Department, involved 13,500 joint service participants, federated forty- 
two models [Koz02] in seventeen locations and nine live training sites, and concluded three 
weeks of playing a classified 2007 scenario with specified objectives [MC02], on August 
15, 2002, after taking two years and $250 million to plan and execute [Nur02, Nay02]. 
Lessons-leamed from MC02 impacting M&S credibility and confidence of results identi¬ 
fied in [Nay02, Sch02b] includes recommendations for a better Department doctrine defin¬ 
ing how to achieve value from the growing portfolio of the Departmenfs M&S, independ¬ 
ent reviews of the experiments, a better definition between objectives designed to validate 
concepts and objectives designed to identify victors of the virtual battles, and an increased 
emphasis on effectiveness, including smaller games and M&S to test and train. 
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D. SUMMARY 

Chapter II provided a literature survey on the eurrent researeh areas relevant to De¬ 
partment simulation eredibility. The literature survey clearly revealed two major eras af¬ 
fecting Department simulation credibility. The Growth Era before 1994 experienced over 
forty years of research, experimentation, and development of a wide variety and types of 
M&S with varying degrees of credibility. The Managed Era, or post-1994 period, intro¬ 
duced policy and management procedures as the Department began to plan, program, de¬ 
velop policy and procedures, budget, and corporately manage the Department Enterprise¬ 
wide M&S portfolio developed during the Growth Era and plan for new joint-simulations 
to replace many of the legacy Growth Era M&S. This chapter also introduced the Depart¬ 
ment’s M&S management framework for establishing credible simulations, including 
classes of simulation, simulation hierarchies, and categories of M&S (live, virtual, con¬ 
structive), within a prescribed hierarchy of M&S (engineering, engagement, mission, and 
campaign). 

The current Department M&S hierarchy maintains only general, qualified levels for 
M&S fidelity and resolution, provides only limited support for the user to ascertain a simu¬ 
lation’s attributes. The model provides little if any support for quantifiable measures of 
fidelity and resolution. In addition, metrics based on this model tend to be qualified, sub¬ 
jective, and lack authority. In this research we analyzed the current Department M&S Hi¬ 
erarchical model as a core asset and reviewed alternatives for mining reverse engineering 
and reengineering these core assets into product line architecture. 

Other Department initiatives included the establishment of new distributed simula¬ 
tion capabilities supporting the expanding need of credibility by the major stakeholders. 
Major categories of stakeholders influencing the Department’s M&S management priorities 
included the acquisition, training, operations, analysis, and experimental domains. 

Research indicated that M&S credibility was a multi-dimensioned problem. Chap¬ 
ter III through Chapter VII review different dimensions of the Department’s M&S credibil¬ 
ity challenge. A taxonomy provided at Eigure 2-1 represents the synthesis of the major ar¬ 
eas identified in the literature search to counter these major challenges and support credibil- 
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ity in M&S and confidence in the results they produce, including the architectural frame¬ 
work, VV&A, software engineering, process improvement, and knowledge acquisition. 


39 



THIS PAGE INTENTIONALLY LEET BLANK 


40 



Ill, CREDIBILITY OF DEPARTMENT SYSTEM REPRESENTATIONS 


A. INTRODUCTION 

Chapter III introduces M&S credibility issues and identifies concerns that decision¬ 
makers have with Department simulation model representation and results, followed by an 
overview of the Departments W&A processes, and several factors contributing to the im¬ 
plementation of, or shortfalls of the VV&A processes in the Department. Two major peri¬ 
ods, the Growth and the Managed Eras characterize the Department’s experiences with 
M&S. The Growth period included the forty years of M&S development prior to 1994, and 
the Managed Era includes the Department’s post-1994 efforts to improve M&S credibility. 

B, THE GROWTH ERA: M&S CREDIBILITY ISSUES IDENTIFIED 

In 1978, a United States Comptroller General report [PAD78] identified a number 
of major systemic quality issues in the simulation software development practices of gov¬ 
ernment agencies and concluded that: 

There are no generally accepted guidelines for establishing models credibil¬ 
ity. In addition, there is not a generally accepted threshold beyond which a 
model can be termed ‘credible’, and the concept of credibility varies greatly 
between model builders/developers and users/decision-makers as well as 
among individuals within either of these two broad groups [PAD78]. 

The need for credible M&S grew in the Nation’s private and public sectors. By 
1980, information from computer-based simulations played an expanded role in the analy¬ 
sis of public policy issues and supported decisions for systems costing billions of dollars 
[Sta74, PAD78, PAD79a, PAD79b, PAD80, Che86a, Che86b, Che87]. Computer-based 
simulations also aided political-military analysis [DBW87, DH92a, DH92b], supported the 
development of strategic course of action development underpinning national policy 
[Lie97], assisted elements of the Department’s decision-making process [HRA+01], and 
recently emerged in an expanded role in the United States National Security Strategy 
[Bus02d]. 

The report Models, Data, and War: A Critique of the Foundation for Defense 
Analysis, [PAD80] found the same major systemic shortcomings in Department-wide M&S 
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cited earlier by the General Aeeounting Offiee (GAO) in other government ageneies in¬ 
eluding the GAO [PAD78], the Federal Energy Commission [PAD79a], and a survey of 
other government ageneies [PAD79b]; 

• Overall poor quality, 

• Laek of doeumentation, 

• Inadequate understanding of model assumptions, limitations, and capabilities, 

• Insuffieient model development praetiees, 

• The need for better eoordination between the model developer and the user, 

• The requirement for better monitoring of development efforts, 

• The need for improved proeedures to update and maintain models, 

• Improve how the M&S eredibility is established, 

• Improve how the validity of the model is determined, 

• Problems with finding data to make the model funetion, 

• Models supporting eredible Department deeisions should be transparent, ap- 
praised^^ and eonsistent [PAD78, PAD79a, PAD79b, PAD80]. 

However, by 1988 the Department realized that the existing ways and means for 
determining simulation eredibility were insuffieient and improving simulation eredibility 
and user eonfidence in simulation results would require new policies, proeedures, re- 
sourees, and organizations. A Secretary of Defense memorandum [Car88] noted that, “as 
the need to improve the eapabilities and eredibility of simulation continues to inerease, we 
are redoubling our efforts to develop eomprehensive, eonsistent and eredible guidanee to 
the serviees” [Car88]. The Direetor, Operational Test and Evaluation, stated that when “ 
results from modeling and simulation are used.. .speeial eare is neeessary to ensure they are 
eredible” [Kri89]. Early evaluations of simulations used in operational tests defined simu¬ 
lation eredibility as a, “fragile eommodity...[that] as applied to the M/S proeesses and re¬ 
sults, is a eombined impression of the inputs, proeesses, outputs, conelusions, the persons 
or agencies involved, and the strength of the evidenee presented” [DOT89]. 

C. THE MANAGED ERA FOR CREDIBLE DEPARTMENT M&S 

In response to the identified systemie defieieneies, the Department implemented 
and updated several M&S management initiatives ineluding M&S management poliey 


The term ‘appraised’ in this context means the model is mathematically correct, matches the real world, and uses empirically valid 
data [PAD80]. 
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[DoD94], master plans [DoD95, DoDOla], and verification^^, validation’^, and accredita¬ 
tion’'^ instructions and procedures [DoD96, GMS+96, DoDOOa, RPGOO, DoDOlb, 
DoD02a], including VV&A of conceptual models, and data’^, in order to resolve these is¬ 
sues. The Department published the first Department M&S management directive, DoD 
Modeling and Simulation Management [DoD94] in January 1994 to resolve systemic is¬ 
sues, provide departmental M&S policy, determine organizational responsibilities, define 
information requirements’^, establish an investment program, and develop procedures, in¬ 
cluding VV&A, to “Strengthen the uses of M&S in the Department of Defense” [DoD94]. 
[DoD94] also assigned Department M&S Executive Agents (MSEAs) responsibilities to 
DoD Components for common” and general-use’^ M&S applications such as terrain or en¬ 
vironment. 

However, major systemic shortcomings persisted to undermine confidence in simu¬ 
lation results when the Department published the first Department M&S master plan, DoD 
Modeling and Simulation Master Plan [DoD95]. At the time, M&S were still too costly, 
failed to meet users needs, remained stove-piped within the functional communities, ex¬ 
perienced excessive development time, lacked credible VV&A procedures, and needed im- 
provements in many quality characteristics: interoperability , security, maintainability, ex¬ 
tensibility, and usability [DoD95]. 

Concurrently, a 1995 Modeling and Simulation Benefits Task force [WSM-l-96] 
completed a limited review of the Department’s corporate M&S assets to document the 
quantifiable benefits of the Department’s investment in M&S. Based upon a very limited 
sample size and meta-analysis, [WSM-l-96] achieved mixed results and concluded that: “a 
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Verification is the process of determining that a model or M&S implementation accurately represents the developer’s conceptual 
description and specification. Verification also evaluates the extent to which the model or M&S has been developed using sound and 
established software engineering techniques [DoD95]. 

73 

Validation is the process of determining the extent to which a model or M&S is an accurate representation of the real world from the 

perspective of the intended use(s) of the model or M&S [DoD95]. 
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Accreditation is the official certification that a model or M&S is acceptable for use for a specific purpose [DoD95]. 
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Data is a representation of facts, concepts, or instruction in a formalized manner suitable for communications, interpretation, or proc¬ 
essing by humans or by automatic means [DoD98]. 

“When possible, existing information systems shall be used or modified rather than creating new systems” [DoD94]. 
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Common-use M&S are applications, including services or materials, provided by a DoD Component to two or more DoD components 
[DoD94]. 

General-use M&S applications are representations used by or common to many M&S, including the physical environment or envi¬ 
ronmental effects such as terrain, atmosphere, or hydrographic effects [DoD94]. 
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Interoperability is the ability of a model or simulation to provide services to, accept services from, other models and simulations, and 
to use the exchanged services to operate effectively together [DoD95]. 
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formal reporting mechanism does not exist for gathering information, nor do methodolo¬ 
gies exist for objectively assessing the value of using M&S” [WSM-l-96]. 

A 1996 DoD Study on the Effectiveness of Modeling and Simulation in the Weapon 
System Process [PHP-l-96] identified three major types of M&S challenges: technical, cul¬ 
tural and managerial, and fifteen specific issues, many previously cited by [PAD78, 

PAD80, DoD95]. Many of the systemic challenges identified by [PHP-l-96] addressed 
credibility concerns with issues including proprietary models and data, security of data, 
availability of data descriptions, interoperability of M&S tools, consistent variable resolu- 
tion descriptions, physics-based models based on empirical data rather than physical prin¬ 
ciples, and continued W&A deficiencies. 

[PHP-l-96] also cited proprietary data, and resistance to funding V&V as two M&S 
management problems in the Department. The Department also acknowledged the limited 
research and review of V&V methodologies within the community and that the “verifica¬ 
tion and validation (V&V) process leading to accreditation is expensive and not well un¬ 
derstood” [PHP-l-96]. In addition, the complexity of the software engineering process of 
non-trivial software-intensive simulations and establishment of M&S credibility [RosOl, 
WPL02] add difficulty to the validation process and may limit credible validation efforts 
[KMOla]. 

These challenges highlighted the lack of “broadly accepted community standards 
for representing military systems and organizations in M&S” [DoD95]. Consequently, rep¬ 
resentations of the same system in different models were frequently incompatible according 
to [DoD95], limiting the effectiveness of the federation objectives and reducing confidence 
in the results as a consequence of unresolved substantive interoperability issues. To im¬ 
prove M&S credibility, [DoD95] established six Department-wide M&S objectives with 
seventeen sub-objectives, and one hundred and fifteen supporting action. The six major 
Department M&S objectives delineated by [DoD95] established requirements to: 


Resolution is the degree of detail and precision used in the representation of real-world aspeets in a model or simulation [DoD95]. 
Resolution: 1. The degree of detail used to represent aspects of the real world or a specified standard or referent by a model or simula¬ 
tion. 2. Separation or reduction of something into its constituent parts; granularity [RPGOO]. 

44 



• Develop a eommon teehnieal framework (CTF) for M&S with sub-objeetives for 
HLA eonceptual models of the mission spaee^* (CMMS), and data standardization, 

• Develop timely / authoritative representations of the natural terrain, 

• Develop authoritative representations of systems, 

• Develop authoritative representations of human behavior, 

• Establish a M&S infrastrueture to meet developer / end-user needs with a VV&A 
sub-objeetive, 

• Share the benefits of M&S [DoD95]. 

Aeknowledging the critieality of eomputer-based information [JBC+99] stated: 

“Software is the physieal structure of the information age... It is fundamental to economic 

success and national security.” However, here may be even more challenges, as the bipolar 

threats of the late twentieth century fade way and are replaced by the modern twenty-first 

century world with a multitude of military asymmetric threats, [Lie97] cautions that it is: 

Now necessary to address a much more confusing set of problems: unex¬ 
pected environments, non-traditional forces, new missions, and new con¬ 
cepts of operation. Relevant data are limited. Many of the widely used 
models of the last 40 years or so may be inappropriate for the new arena 
[Lie97]. 

Trying to overcome over forty years of modeling and simulation history without 
V&V guidance, while maintaining a large inventory of legacy models, and attempting to 
stay ahead of the learning curve, the Department developed the first VV&A instruction, 
DoD Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A) 
[DoD96], and VV&A best practices guide. Department of Defense Verification, Validation, 
and Accreditation (VV&A) Recommended Practices Guide [GMS+96]. In addition the De¬ 
partment continuously updated M&S policy [Gan98, Gan99, WolOl], verification and vali¬ 
dation instructions [DoDOOa, DoDOlb, DoD02a], masterplans [DoDOla, DoD02c, 
DoD02d], manuals [DoD98], verification and validation technical guidance [GMS+96, 
RPGOO] and developed a strategic plan [DoDOSa] to support these objectives. Department 


Designated as the Conceptual Model of the Mission Space (CMMS) in the [DoD95]. The tenn CMMS is in the process of officially 
being changed to the Functional Description of the Mission Space (FDMS) [RPGOO], 
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components also established supplemental policies and proeedures [AP93, AR97, 
DON99, AFIOO], 

The Department’s hrst strategic plan [DoD03a], provides six strategic goals with 
twenty-nine objectives supporting the new M&S mission and vision statements to help the 
Department conduct war, transform the foree, strengthen, Joint warhghting capabilities, 
develop and test new coneepts to: 

• Operationalize M&S as a real-time deeision support tool for the warfighter (five 
supporting objeetives), 

• Provide rapidly and readily aeeessible and exeeutable M&S (six supporting objee- 
tives), 

• Inerease M&S awareness, education, training, and collaboration (four objectives), 

• Effeetively represent the complexity of modem warfare aeross the full speetmm of 
operations (five supporting objeetives), 

• Employ M&S to aceelerate aequisition, reduce life-cycle-eosts, foster interoperabil¬ 
ity, and improve quality of systems (five supporting objectives), 

• Align and increase scienee and teehnology investment to transform M&S for the 
warfighter (four supporting objectives) [DoD03a]. 

The Department and the Service components also developed M&S guidelines and 
best praetices [PMR94, GMS+96, Eal97, SAEOO] to improve eonfidence by decision¬ 
makers that Department M&S provides reasonable, correct, defendable and credible results 
for given situations, oireumstanees and environments. The private and teehnieal seetor, 
including the Simulation Interoperability Standards Organization (SISO), the International 
Standards Organization (ISO), and the IEEE also eontributed to the growing M&S body of 
knowledge. In addition to the development of policies, plans, procedures, instmctions, and 
guidanee the Department established the Defense Modeling and Simulation Office 
(DMSO) to plan, program, coordinate and budget the Department’s M&S program and an 
Exeeutive Council for Modeling and Simulation (EXCIMS) to provide assistance and ad- 
viee on major M&S issues [DoD94]. 

The Department also made signifieant investments to improve the credibility in 
M&S during the 1990s. These major investment initiatives includes the common technical 
framework initiatives for HEA, development of a HEA-eompliant Runtime Infrastructure 


DoD Components include the Office of the Secretary of Defense (OSD) Components, the Military Departments, the Chairman of the 
Joint Chiefs of Staff, the Combatant Commands, the General Counsel of the Department of Defense, the Inspector General of the De¬ 
partment of Defense, the Defense Agencies, and the DoD Field Activities [DoD94]. 
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(RTI), authoritative data sources, including the Environmental Effects for Distributed In- 
teractive Simulation (E DIS) program [PBG+94], and the Synthetic Environment Data 
Representation and Interchange Specification (SEDRIS) project, and major joint simulation 
development efforts such as the JSIMS, JWARS, and JMASS initiatives. 

D. ESTABLISHING DOD M&S CREDIBILITY AND USERS CONFIDENCE 

The Department established the M&S VV&A process to develop credibility in 
M&S software and maintain user confidence in the results. The Department and the private 
sector M&S community developed a body of research, studies, best practices, and analysis 
focused on improved verification and validation theories, methods, techniques, tools, and 
procedures to remedy systemic issues and shortcomings identified by reports, studies, and 
other assessments cited earlier. 

Research into the Department’s W&A process identified several patterns between 
the individual W&A processes and several other areas affecting the establishment of 
M&S credibility. The areas identified in Figure 3-1 address topics consistently raised as 
concerns and issues in the VV&A literature and provides a synthesized M&S credibility 
taxonomy addressed in the following chapters: 

• The Department’s verification process, 

• The Department’s validation process, 

• The Department’s accreditation process, 

• The role of data in the VV&A process, 

• Verification and validation techniques, 

• Shortfalls of the Department’s VV&A process, 

• Software development / software engineering practices, 

• The Department’s M&S framework, 

• Heterogeneous data, 

• VV&A practices, 

• Organizational areas and issues, 

• The Department’s Common Technical Framework, 

• VV&A techniques, 

• Software architecture, 

• Quality, 

• Process Maturity, 

• Secondary contributing factors (e.g. reuse, documentation, risk management, cost, 
return on investment). 
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Department institutional eonstraints. 





Figure 3-1. Synthesized M&S Credibility Taxonomy 

E, VERIFICATION, VALIDATION & ACCREDITATION PROCESSES 

Early studies by [Arm64, Mee64, and NF67] identified the lack of verification 
methods in the M&S process and proposed the concepts of high face validity, conceptual 
validity, and operational validity. Later, [Mih72, Sha75] addressed the concepts of V&V in 
conjunction with systems analysis, system synthesis and model analysis. [Sar91, Sar92, 
Sar98] considers M&S V&V critical and addressed model validation methods and tech¬ 
niques including conceptual model validity, model verification, operational validity, data 
validity and development of a “model range of accuracy” from techniques employed in 
M&S output analysis. 

[MBZ99] provided a concise chronology of M&S credibility from 1962 through the 
1990's and identified the growing emphasis and research on V&V, and the growing aware¬ 
ness of the importance of the underlying software development foundation. In 1988, the 
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Director, OTE acknowledged the emerging efforts to establish Department M&S verifica¬ 
tion, validation and credibility assessments, noting that it was “essential that the M&S em¬ 
ployed and the results derived from them be credible” [Kri88]. 

There is a significant body of related literature on improving M&S credibility and 
improving confidence in simulation results. A large body of simulation-related topics in¬ 
cluding risk reduction, V&V principles and techniques, M&S quality characteristics, proc¬ 
ess improvement, software development / engineering procedures, domain implementation 
uses, and approved architectural standards for achieving higher M&S confidence levels, 
includes research by [BL91, Sad91, Dow92, Gia92, Hen92, Met92, Bec94, Lew94, Par94, 
Cau95, Gan95, BCN96, CCK+96, Por96, FY97, ML97, Mue97a, Mue97b, Pre97, Bar98, 
Che98, Gan98, Gau98, HM98, JDD98, LR98, MT98, NM98, NMY+98, Pac98c, Roa98, 
ROG98, BSW99, Dah99, DMS99, FGW+99, GB99, Gre99, Mar99, Pac99a, San99, 

Whi99, BFOO, BSF+00, COHOO, EKHOO, HNC+00, MFU+00, SanOO, BSOl, CooOla, 
CYOl, DDHOl, FEDOl, FHOl, HalOlb, GKBOl, HNK+01, FSOl, FuqOl, MJFOl, MorOl, 
PacOlb, PacOlc, PugOl, RosOl, YPHOl, BMF02, Bre02a, CW02, Dan02, FRC+02, Kil02, 
Pac02b, Pac02c You02a, You02b]. 

More recently, [CoyOOb and BGOl] noted several significant systemic M&S chal¬ 
lenges within the DOD Test and Evaluation community affecting confidence in the results. 
[CoyOOb] specifically cautioned that the use of M&S to extrapolate beyond the known 
bounds almost never works, however, interpolating between two demonstrated test results 
is an acceptable practice in evolutionary acquisition programs. [Tho97a] also raised the 
concern that the V&V process lacked adequate definition as recently as 1983. 

Current V&V theories, methods, techniques, and procedures cover a wide range of 
activities cited by [Kri89, RSH90, Rit92, WS92, WF93, ARS+96, JAS97a, JAS97b, 
Pac98d, RPGOO, Roa98, SMOO, RakOl, WriOla, WH02] and techniques proposed by 
[Gia92, Cau95, FHOl, FriOl, GMS+96, KFC98, MM98, RPGOO]. Verification and valida¬ 
tion for large complex M&S such as mission-level models or physics based models have 
additional impediments [KMOla] including the dynamic nature of M&S evolution, the cost 
and constraints of the data, the difficulty of validating the models, and the limited shelf life 
of previous validation efforts. 
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The Department published [GMS+96], and established the twelve guiding princi¬ 
ples for the W&A of M&S and federations. [Bal98] also contributed fifteen guiding prin¬ 
ciples to provide better insights into W&A activities, supporting the Department’s model 
[GMS+96] or the [Bal98] M&S life cycle models. A follow-on Internet variant of the 
[GMS+96], the W&A Recommended Practice Guide [RPGOO], promotes the current best 
V&A practices and procedures for legacy M&S (Figure 3-2). 


PROBLEM SOLVING PROCESS 



Define 
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Figure 3-2. W&A and Legacy M&S (From [RPGOO]) 

Other approaches evolved. [RakOl] identified three basic processes for software 
validation based on testing, measurement, and software reliability growth. Current W&A 
procedures built on these theories and lessons-leamed from previous W&A approaches, 
including the Aggregate Level Simulation Protocol (ALSP) W&A process [TP96] and the 
Distributed Interactive Simulation (DIS) W&A process [GW96, MBZ99]. [Bal98] ad¬ 
vanced an analytic hierarchy process, which supported the measurement and evaluation of 
both quantitative and qualitative factors, expert knowledge, independent evaluation and 
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comprehensive assessment to improve user eonfidenee in M&S results. In addition, 

[Bal98] introdueed a model-testing eoneept ealled model verifieation, validation, and test 
(VV&T) for identifying and aseertaining whether specifie inaceuraeies or errors exist in the 
model. [Bal98] also reeommended eontinuous V&V planning, execution, and documenta¬ 
tion throughout the entire M&S life eycle. 

A Modeling and Simulation Information Analysis Center report on VV&A auto¬ 
mated support tools [DMSOOb] and a subsequent artiele by [GFMOl] proposed leveraging 
existing software development industry tools, basieally expanding the development and 
applieation of V&V automated test support tools analogous to those eurrently employed in 
the test and evaluation eommunity. [Lew98] proposed a V&V Managers Toolkit based on 
a VV&A management model with a metrie suite for eost, performanee, sehedule, and teeh- 
nieal performanee indicators. Currently, automated tool support for the VV&A proeess is 
extremely limited [JorOl]. In addition, [HNP97] eautions that automated tools to assist the 
eomplex and diffieult validation proeess may be undesirable from an engineering view¬ 
point. 

Whether using qualitative or quantitative methods, a eombination of the two meth¬ 
ods to eomplement eaeh other, with or without automated tools, the V&V proeess requires 
disciplined proeesses and proeedures to provide satisfaetory levels of eonfidenee in the 
simulation results. However, at the present time [RakOl] notes, “basie software V&V ae- 
tivities are not well understood and are applied ineonsistently” [RakOl]. 

1, The Department’s Verification Process 

The software V&V proeess entails a broad seope of aetivities, whieh support soft¬ 
ware quality eharaeteristies [DB91, Dav92e, AV98, HalOla] diseussed in Chapter VI. The 
verifieation proeess determines the M&S eapability and establishes the foundation for eon- 
fidenee in the model, establishing quality early in the development proeess by ensuring the 
developer aecurately captured the validated eoneeptual model in the implementation de¬ 
sign. “Verifieation,” as defined by the Department is “the proeess of determining that a 
model implementation aecurately represents the developer’s eoneeptual deseription and 
speeifications” [DoD94, DoD96, DoDOla] when eompared with the first abstraction of the 
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real world, the eoneeptual models of the mission space (CMMS). In addition, “verification 
ensures that a M&S meets all the requirements specified by the user and that it implements 
those requirements correctly in software” [GMS+96]. 

There are many other definitions for verification. Verification is also concerned 
with solving the equations right [Roa98]. The IEEE similarly defines verification as “con¬ 
firmation by examination and provisions of objective evidence that specified requirements 
have been fulfilled” [IEE98b]. Verification also supports the implementation of the func¬ 
tional interface, and output requirements and may be defined as “Did I build the thing cor¬ 
rectly” [GMS+96]? 

The conceptual model in the M&S process is analogous to the keystone in a build¬ 
ing archway. A missing, incorrect, incomplete or sub-standard keystone threatens the 
structural integrity of the archway. Today, [Pac99b] notes conceptual model documenta¬ 
tion “for many legacy M&S is limited or non-existent”[Pac99b]. [Ham96, HNP97] further 
note that confidence in the verification process depends upon the underlying model’s valid¬ 
ity. The absence of a valid conceptual model threatens the integrity and credibility of the 
simulation and confidence in the simulation results. 

2. The Department’s Validation Process 

Model and simulation validation is performed at two distinct levels: 1) conceptual 
model validation, and 2) result validation, and satisfies the criteria of how well the func¬ 
tions match the real world and addresses “Did I build the right thing”[GMS+96]? The con¬ 
ceptual model validation is an essential, integral component of the Department’s W&A 
process and is the foundation for establishing credibility, defining fidelity requirements and 
developing accurate model representations. Results validation compares the M&S results 
with known or projected behavior, tests, and sensitivity analysis or subject matter expert 
opinion to determine if the results are “sufficiently accurate for the range of intended uses 
of the M&S” [GMS+96]. 

The validation process complements and supplements the verification process, and 
supports development of M&S credibility [IEE86, Sar91, Cyn92, DH92a, Hen92, Sar92, 
Ham96, Bal97, HNP97, IEE97c, Bal98, IEE98b, Sar98, Pol99, CooOla]. Model and simu- 
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lation validation “is the process of determining the degree to which a model is an accurate 
representation of the real-world from the perspective of the intended uses of the model” 
[DoD94]. [Roa98] views the validation process as solving the right equation. The De¬ 
partment’s definition of validation [DoD94] includes the validation of conceptual model, 
results, and data in the V&V process supporting multiple Department domain requirements 
including acquisition programs [CS97, COHOO, CoyOOb]. The IEEE defines validation is 
as the ’’confirmation by examination and provisions of objective evidence that particular 
requirements have been fulfilled” [IEE98b]. 

There are many validation strategies, with several noted in this research. [HNP97] 
identified two possible strategies to validate a model: the axiomatic approach (e.g., theory- 
driven) where the existence of a set of assumptions describe the fundamental truths of the 
problem domain, and the model proven as a theorem; and the empirical approach (e.g., 
data-driven) in which the model performance is compared to expected values or historical 
data. Generalized validation (e.g., evaluation) according to a VV&A taxonomy developed 
by [Dav92b] would include empirical and theoretical evaluation methods in addition to 
evaluation by other methods. Output accuracy methods outlined by [Kil02] includes 
benchmarking, face validation, results validation and sensitivity analysis. According to 
[Kil02] there are three primary ways to determine output accuracy: another M&S or 
benchmarking, subject matter expert experience employing face validation, results valida¬ 
tion with test data, with all methods sometimes supplemented with sensitivity analysis. In 
order to achieve the highest degree of M&S credibility, [EKOO] opine that the “most defini¬ 
tive test of a M&S model’s validity is to establish that it’s output data closely resemble the 
output data that would be expected from the actual (proposed) system” [EKOO]. 

The degree of confidence in M&S depends heavily on M&S anchoring in physical 
testing [WelOO]. Although tests and evaluations may develop sufficient confidence levels 
that the model is valid for its intended purpose, [Bal98] cautions that complete M&S model 
testing is not possible and that successfully testing each sub-model (module) does not im¬ 
ply overall model credibility. Eurthermore, [EKOO] acknowledges the timing and relation¬ 
ship of validation is critical for establishing and improving confidence levels that “correct” 
results are available for credible decisions. Eive key quality factors identified by [MEWOl] 
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support M&S credibility: capability, software accuracy, data accuracy, results accuracy and 
usability. 

[BCE+99] addresses another difficult testing challenge of striking a balance be¬ 
tween live testing, sound analytical methods and ground testing. M&S is extremely chal¬ 
lenging for advanced or unprecedented systems, where “substantial design deficiencies or 
unanticipated system characteristics requiring correction continue to be revealed during 
flight tests despite the sometimes substantially and costly applications of M&S. ..even those 
subjected to comprehensive verification and validation” [BCE-l-99]. 

Even with these concerns, there are still differences of opinion within the M&S 
community today in the requirement for V&V, the scope of the effort, the high cost, how it 
is implemented, specific techniques, and determining “how much V&V is enough?” In ad¬ 
dition, [Bal98] suggests that some tests are more appropriate to evaluate the behavioral ac¬ 
curacy or validity of the model, while other tests better determine the accuracy and verifi¬ 
cation of model transformation from one form into another. Consequently, the Depart¬ 
ment’s M&S domain “lacks a coherent process that links V&V information to application- 
specific requirements for M&S credibility”[GMS+96]. 

3. The Department’s Accreditation Process 

The final action of the Department’s VV&A process is the decision-makers accredi¬ 
tation^^ [SS92, CSC94a, CSC94b, JAS96, San97a, Sta97, PacOla] of the simulation. M&S 
accreditation is “the official certification that a model, simulation or federation of simula¬ 
tions is acceptable for use for a specific purpose”, implemented by the Department 
[DoD95, GMS+96] to provide credibility and confidence in M&S developed for another 
purpose or agency. This process provides the decision-maker with an understanding of the 
capabilities and limitations (e.g., caveats) of the simulation. Accreditation in the Depart¬ 
ment’s M&S domain requires confidence in the V&V process, documentation, data, use 
history, and known limitations of legacy simulations, especially when developed for other 
purposes, or maintained by different organizations. 

Accreditation in terms of information system security has another meaning: The formal declaration by a Designated Approval Author¬ 
ity (DAA) that an information system is approved to operate in a particular security mode using a prescribed set of safeguards at an ac¬ 
ceptable level of risk (DITSCAP) [MonOOb] 
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The challenge of achieving credibility in the Department’s M&S was succinctly 
noted by [San97a] who wrote that it “is not confidence in M&S we seek, but confidence in 
the decisions we are making” [San97a]. [SS92] propose that all models are wrong in some 
aspects, and the validation process identifies how and when the model is erroneous, while 
the accreditation process is the “determination that the decision to be made is not sensitive 
to these errors and limitations”[SS92]. As emphasis on formal VV&A grew within the De¬ 
partment, two kinds of accreditation were initially proposed according to [PacOlc]; 1) class 
or domain accreditation, as a form of blessing of the simulation for a kind (e.g., class) of 
application; and 2) specific application accreditation, as a formal certification that the simu¬ 
lation and its associated inputs is appropriate for a particular use. [PacOlc] cautions against 
the temptation to use class accreditation, noting that the potential exists today within the 
Department’s M&S domain to “use... almost right, but not quite right software”, contrib¬ 
uted to the uninsured five billion dollar loss of the Ariane 5 booster in 1996 [LLF-l-96]. 

Department component accreditation responsibilities include the accreditation of 
forces and capabilities employed in joint, general, and common-use M&S, and ensuring the 
appropriate representation of its forces and capabilities in joint and common use M&S de¬ 
velopment [DoD94]. Simulation credibility defined by [Kil02] includes the ability of the 
most important elements to meet the required capability and intended use. Required capa¬ 
bilities explained by [Kil02] include descriptions of the simulation’s purpose, represented 
model entities and fidelity, physical environment description, element relationships, and 
assumptions and limitations. Addressing accreditation of federations, [PacOla] identifies 
the compatibility of individual federates in a federation as a key element in validation as¬ 
sessment to determine if the federation can appropriately support the intended application. 

4. The Role of Data in the W&A Process 

Data is critical to the Department goal of improving credibility and confidence in 
M&S results, and plays a key role in developing simulation credibility and underpinning 
user confidence in the results. The [RPGOO] cites four basic types of data including: 1) ref¬ 
erence or metadata, 2) hard-wired data such as constants and set parameters, 3) instance 
data for input and output functions, and 4) validation data comprised of actual measure- 
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ments or real-world facts used for comparison with M&S results for validation. The three 
basic metadata types: administrative, descriptive, and quality, identify databases [Gor96], 
or individual data elements [RPGOO], whereas, metadata quality supports the quality as¬ 
sessment techniques, results, and addresses attributes for accuracy, precision, complete¬ 
ness, portability, consistency^"^, currency and flexibility. An additional fifth type of data 
includes exchange metadata for federations. Multiple authors including [Atw91, NMY+98, 
HSB98, GozOO, PFL+00, DelOO, PKLOl, SHOl, TolOl, LWK+02] address data verifica¬ 
tion, validation and certification issues. 

From another viewpoint, [HNP97] suggests three categories of data: discrete data, 
which may take on one of a finite set of values; continuous data, which may assume any 
value in an interval of real numbers; and categorical data, which may take on a finite, nor- 

oc 

mally small set of values. Whatever data ontology is chosen, authoritative data is re¬ 
quired for developing credible representations and other critical activities in Department 
M&S. However the [DoDOla] cites authoritative data as “ the single-most pervasive defi¬ 
ciency area identified with M&S use” [DoDOla]. 

Data is also critical to the Department’s VV&A process, yet widely misunderstood 
and sub-optimally executed in the Department. [Sar98] cited historical data validation 
methods such as rationalism and empiricism, as well as a predictive validity approach to 
improve the process. Acquiring authoritative, certified data supporting sufficient M&S 
validation processes is also difficult, and exacerbated by the existence of scientific and 
technical data characterized as complex data^^. Complex data categories include highly 
derived data, objects, compositions and artifacts of legacy systems [DoD95]. 

A major Department study on the existing data verification, validation and certifica¬ 
tion (VV&C) methodology, [NMY+98], concluded that data quality results from the inte¬ 
gration of the data producer’s V&V process, which supports the user’s data V&V efforts. 
Both activities are components of the M&S V&V process, conducted during the entire 
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Consistency refers to data that is maintained free from variation or contradiction [DoD98]. 

Authoritative data comes from authoritative data sources whose products have undergone producer verification, validation, and certi¬ 
fication activities [DoD98]. 

Complex data cannot be characterized as a single concept or data element [DoD95] 

Highly derived data includes categories such a probability of hit/kill. Objects employ multiple inheritance, multiple root hierarchies 
and polymorphic attributes. Composition includes images, binary large objects (blobs), and compound documents. Artifacts are nor¬ 
mally attributed to data from legacy systems designed to reduce storage requirements by the use of embedded codes or logic [DoD95]. 
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M&S life cycle, with data quality metadata bridging the two activities. However, 
[NMY+98] identified the lack of discrete, formal VV&C standards supported by data qual¬ 
ity metrics, including metadata quality metrics as a systemic Departmental data deficiency. 
Four major deficiencies noted by the [NMY+98] study included: 

• Data V&V methods lacked consistency, 

• No linkage between the development of data products and user V&V requirements, 

• No central repository for data V&V information, 

• Data VV&C not synchronized with the M&S VV&A effort [NMY+98]. 

The concept that data V&V should be an integral part of the M&S life-cycle proc¬ 
ess [IEE96, IEE97b] as opposed to a separate distinct procedure has been emphasized by 
recent research [Sar98, NMY+98], and reflected in Eigure 3-3. This process is a variant of 
the earlier recommended procedure in the [DoD95] guidelines defining a separate process 
for verification, validation and certification (VV&C) of data by the producer and the user. 
The separate VV&A and VV&C processes raised the probability that validation of the of 
data for the intended use was more implied than the result of an explicit, defined, repeat- 
able process. 

Several methodologies exist to improve data quality. Data quality [DoD98] is the 
correctness, timeliness, accuracy, completeness, relevance, and accessibility characteristics, 
which make data appropriate for use.^^ Validated data for rapidly evolving M&S is a major 
challenge, but is essential for M&S tools if they are to achieve their full potential. Data 
accuracy cited by [Kil02] addresses the appropriateness of the data, including resolution; 
the quality of the data established by the data producer and user; and the correctness of data 
transformations. [Cau95] proposes a reduced order metamodel validation methodology to 
compare M&S results with real-world data, if available. Suggested improvements in the 
data validation process supported by [HHOOa, HSOOb] address data interchange formats 
(DIE), while [HSB98] recommend a CMMS data dictionary for improved interoperability. 
[DoDOla] adds the requirement for new V&V methodologies and improved technologies, 
including software quality metrics detailed by [Kan03] to enhance the software quality 
V&V practices within the Department. 

Quality statements are needed for source, accuracy (positional and attribute), currency, logical consistency, completeness (feature and 
attribute), security classification, and releasibility [DoD95] 
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Figure 3-3. Data VV&A & Legacy M&S (From [RPGOO]) 

[Dav92b] reinforced the need for improved life-cycle VV&A practices with rec¬ 
ommendations to develop positive incentives including a vision for VV&A, and the devel¬ 
opment of consistent policy and procedures across the Department. [Dav92b] further rec¬ 
ommended early successful implementations, tailoring of VV&A to the corporate culture, 
establishing independent reviews, providing long and short-range plaiming processes, and 
understanding project/program concerns with W&A. [JorOl] notes that data V&V should 
be performed as an integral part of the V&V process, stating that “algorithms only work if 

the required data is available and is accurate for the aggregation portrayed” [JorOl]. 
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Establishing credibility and consistency according to [HBM96] requires the publi¬ 
cation of the approach, data, and tests be for peer review to improve the current situation of 
too many redundant databases, too little testing, too few reviews of data sources, and insuf¬ 
ficient analysis of embedded assumptions. Recent works by [MSOOl] on missing data 
techniques and [SC02] on sparse data method have addressed some of the issues of sparse 
or missing historical data. However, at the present time, the Department’s objective for an 
over-arching coherent program for affordable, timely, verified, validated, and authoritative 
data on demand is still more a goal than a reality [CraOlb]. 

5. Verification and Validation (V&V) Techniques 

The Department’s V&V process for M&S is the approved foundation for achieving 
quality and developing M&S credibility. A current Department M&S objective includes 
maintaining and enhancing the V&V program for models, simulations and data to provide 
users with confidence in the results and knowledge of the limitations. The Department has 
over seventy-five software engineering and model specific techniques for performing V&V 
exist today, with methods ranging from informal to formal methodologies [GMS+96, 

Pol98, RPGOO]. Figure 3-4 identifies the informal, static, dynamic and formal V&V tech¬ 
niques. 

The mathematical and logical formalism increases from the informal techniques on 
the left to the formal methods on the right, as does the complexity of the techniques. In¬ 
formal V&V techniques noted in Figure 3-4 lack rigorous mathematical formalism. These 
tools and methods employing human reasoning and subjective techniques may be very ef¬ 
fective if applied using well-structured approaches, formal guidelines and mature, disci¬ 
plined processes [GMS+96]. Informal V&V techniques are among the most common, fre¬ 
quently used approaches in the Department [RPGOO]. 

Static V&V techniques cited in Figure 3-4 provide insights into the models struc¬ 
ture, modeling techniques, data, control flow and syntax. Different V&V approaches ad¬ 
dressed by [Pac02b] depend on whether the M&S source code is available. V&V agents 
employ these techniques to review the accuracy of the model design and source code. 

These techniques are widely used and do not require model execution runs. Quality auto- 
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mated support tool requirements identified by [AWQ+93, ACC+94, MBH99, GFMOl] are 
also necessary if the Department is to realize the goals of improved results from VV&A 
activities. Few current automated tools are available to assist in this process, except the 
language compiler or interpreter [GMS+96, RPGOO]. 
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Figure 3-4. V&V Teehniques (After [GMS+96, RPGOO]) 

Dynamic V&V Techniques provided in Figure 3-4 evaluate execution behavior of 
the model during run-time. One technique inserts probes or stubs into the executable code 
to collect information during the running of the M&S. This method normally employs 
three steps: inserting the probe or stub instrumentation, executing and collecting the re¬ 
quired data, and analyzing the output file for discrepancies or faults. Dynamic methods 
and statistical techniques have been addressed in a significant body of Department V&V 
guidelines [GMS+96, RPGOO]. [RSH90] submit a propagative approach to sensitivity 
analysis in large-scale M&S where the sheer numbers of parameters make it impractical to 
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test more than a small fraction of relevant cases, a common challenge in all large software¬ 
intensive systems. 

Figure 3-4 lists a number of various statistical V&V techniques available to validate 
models. The best environments for these techniques include completely observable models 
and provide for the collection of all data for validation. In this category, [NF67] advanced 
the technique of Analysis of Variance, while, [MM98] suggest peer-reviewed statistical 
techniques for effectively managing model fidelity. In addition [MM98] contend that De¬ 
partment models often comprise a set of sub-model requiring additional integration into a 
coherent representation of a mission space. 

The literature also reveals an abundance of additional V&V techniques and proce¬ 
dures. [Ham96, HNP97, Bal98, LKOO] discussed simultaneous confidence levels, confi¬ 
dence levels and model range of accuracy to address operational validity. [Ham96, 

HNP97, LKOO] also explained several statistical procedures (inspection, time-series ap¬ 
proaches) for comparing real-world data to M&S output. [ROG98] proposed a multistage 
validation framework consisting of conceptual and operational validation employing two 
statistical methods, a two-sample t test and a two-dimensioned, two-sample Kolmogorov- 
Smimov test [HNP97], concluding that a model valid for one level of detail may be invalid 
for another use. [Ham96, HNP97] suggests the validation process is ambiguous, complex, 
and difficult, subject to undesirable subjective approaches, in part because the first part of 
the validation process builds upon a credible validation of the conceptual model, itself an 
abstraction. 

In addition, [FriOI] explains the application of Fisher’s Combined Probability Test 
and Goodness-of-Fit Tests to support formal statistical comparisons, even when “the data 
are sparsely distributed across a highly dimensional factor space” [FriOl]. [HalOla] sug¬ 
gests that the breakdown of the model into testable functional elements supports validation 
data requirements and makes the point that data “obtained from testing in this manner is at 
a low enough level in the model to support statistical comparisons between model predic¬ 
tions and test results” [HalOla]. Classical statistical techniques [LKOO] that work well with 
ideally distributed, independent observations, may prove difficult to apply correctly with all 
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M&S situations since both real-world systems and M&S output produee non-stationary dis¬ 
tributions and auto-eorrelated observations [RPGOO]. 

Diseiplined verifieation, validation, and aeereditation techniques for M&S and data, 
support interoperability for new large-seale, eomplex simulations, and federations. A sur¬ 
vey of eurrent M&S model VV&T techniques [Bal98] apply throughout the system life cy- 
ele. Several additional methods and teehniques to improve validity preseribed by [HNP97, 
LKOO] inelude: eolleet high-quality data, maintain an assumption doeuments, eonduet 
struetured walk-through, validate model using quantitative teehniques, validate the output, 
and use animation to improve user understanding of the system. [BML02] further suggests 
a theoretieal framework for determining the eonfidenee in a simulation based on the eredi- 
bility of three factors: the model and its embedded data, runtime data, and the operation or 
application of the model. 

[Sar98] identified operational validity techniques for M&S test programs, which in¬ 
clude event validity, eomparison to other models, degenerate tests, extreme eondition tests, 
fixed values, faee validity, parameter variability-sensitivity analysis, traees and Turing 
Tests. [Sar98] also proposes hypothesis tests, whieh support a eomparison of means, vari- 
anees, distributions, and time series of the output variables to determine if the model’s out¬ 
put behavior has an aeeeptable range of aeouraey. However, the eurrent state of M&S 
software development maturity largely laeks objeetive V&V proeesses with quantitative 
methodologies supporting the suffieieney of a model for a speeific purpose. 

The highest possible eonfidenee-levels aehievable in the V&V proeess inelude the 
formal V&V teehniques noted in Figure 3-4. Formal methods are also the most eomplex, 
least understood, and least implemented V&V teehniques in the Department [GMS+96]. 
The seope of formal method teehniques, proeesses, and proeedures inelude [LG93, LGB94, 
FMG94, GMS+96, LG97, Ber98a, Ber98b, LBOO, RPGOO, MLJOl, MJLOl, POPOO, 

BC02], ineluding the ehanges of formal methods [Rob98] employ mathematieal proofs for 
eorreetness, and require signifieant effort. Formal methods applied early in the develop¬ 
ment proeess, normally achieve the best results. However, formal methods applied to the 
most eritieal eomponents, may prove effeetive at any time during the development proeess 
[Mie03]. 
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[NogOO] developed a formal method to automatieally assess risk and the duration of 
a software development projeet supported by evolutionary software proeesses. For large- 
seale software engineering projeets, [BL91] proposed a formal methods mathematieal ap- 
proaeh as a basis for preeise eommunieation and the foundation to aehieve a high degree of 
automation direetly supporting inereased eonfidenee in the software quality. NASA also 
realized the inereasing eomplexity and eritieality of software applieations required an em¬ 
phasis in formal methods. Major faetors identified by [NAS97, CVL+97, CKM+98] for 
inereased use of formal teehniques ineluded the following; 

• Fault-proteetion and safety funetions required software features beyond the hard¬ 
ware level to deteet and isolate failures, and initiate reeovery proeedures, 

• Software-failure eharaeteristies were different than those found in hardware, 

• Complex systems plaeed ever-growing demands on verifieation, 

• Diseiplined software engineering organizations had developed fine-tuned verifiea¬ 
tion proeesses, but defeets still remained in the produet, 

• Few existing teehniques were rigorous enough for improving the quality of produets 
during the early life-eyele phases of requirements generation and design 
[CKM+98]. 

The overlapping tools, teehniques, proeedures and terms of referenee between the 
M&S domain and emerging software engineering/development domains eontinually mani¬ 
fest themselves in the literature. [MLU+00] propose an ineremental qualifieation approaeh 
to M&S VV&A, noting that M&S “is subjeet to software development.. .as well as system 
engineering praetiees. [MLU+00]” [ARS+96] also identified V&V as a system engineer¬ 
ing diseipline, whieh must be ineluded in the domain engineering proeess to improve the 
level of eonfidenee in eritieal software, speeifieally safety and mission-eritieal software. 

The Joint Aeoreditation Support Aetivity (JASA) developed and eontinuously re¬ 
fined the Suseeptibility Model Assessment and Range Test (SMART) methodology 
[JAS97a, JAS97b]. The SMART methodology involves a four-phase approaeh to W&A 
for aequisition M&S ineluding: ineremental reviews, analysis, evaluation, testing, and 
doeumentation of M&S and assoeiated data to support the verifieation and validation proe- 
ess detailed by [CSC94a, CSC94b]. With over ten years of VV&A related-experienee in a 
wide range of M&S programs, the JASA identified eight major obstaeles to VV&A sueeess 
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similar to the systemic issues adversely impacting Department M&S credibility identified 
earlier by [PAD78, PAD79a, PAD79b, PAD80]: 

• Lack of cooperation from the M&S developer based on concerns that identification 
of errors could adversely impact future business, intellectual property [IEE99b] is¬ 
sues, and failure to specify V&V in contract tasks, 

• Eailures to follow disciplined processes, such as configuration management, and 
develop quality documentation, 

• Eailure to define intended use and M&S requirements, including functionality and 
fidelity, 

• Eailure to define accreditation information requirements, 

• Eack of or mismatch of skills, 

• Insufficient priority applied to VV&A efforts, 

• Eack of interpersonal skills demonstrated by VV&A team members, 

• Eack of perseverance [JAS97a, JAS97b]. 

The private sector contractors supporting Department M&S initiatives experienced 
the same systemic issues establishing M&S credibility and maintaining users confidence in 
M&S results. A 1993 Department audit report [RSV+93] of a sample of defense contractor 
M&S programs identified costly duplication, redundancy and unnecessary proliferation of 
existing M&S capabilities with the following quality indicators: 

• Only twenty percent of the defense contractor M&S had completed a formal VV&A 
program, 

• Eive percent of defense contractor M&S had adequate configuration management 
controls in place, 

• Thirty five percent of the defense contractor M&S had adequate documentation, 

• The findings indicated “a substantial lack of effective management control over the 
development and use of M&S by the government contractor community” 

[RSV+93]. 

6. Shortfalls of the Department’s W&A Process 

There are also many V&V shortfalls, limitations, and critiques of the Department’s 
V&V process cited in the literature including [Che87, Dav92b, LA92, Met92, Pay96, 

TP96, CVL+97, KP97, HM98, ME97, Mue97a, NAS97, JDD98, Lew98, NM98, ROG98, 
Pac99b, MEU+00, TRWOO, CraOlb, HNK+01, JorOl, MT98, HS02, Kil02, Pac02b]. 

There are also limits to the confidence established by validation with well known validation 
limits [Kil02] including the limited scope of validation tests, costly validation data which is 
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difficult to obtain, and limited tools [Lew94, Bar98, DMSOOb]. In addition, conventional 
methods and techniques may not be adequate to validate some highly aggregated M&S 
(e.g. mission and eampaign-level models) [Kil02]. 

High-fidelity models, HLA federations [HLA98], and legaey M&S issues [LBSOl, 
ZB02] pose additional, significant challenges for the proper execution of V&V. Effective 
and effieient validation of all potential logie paths in the M&S is probably technically and 
economically infeasible, since the number of possible system states in a large simulation is 
very large. The number of states grows exponentially with regard to the number of degrees 
of freedom in the model simulation or software [Mic03]. [Pae02b] suggests that the utility 
of M&S is limited if the M&S validity and fidelity cannot be adequately quantified. This 
includes testing and test support V&V issues identified by [Kri88, HN97, Che98, Kec99, 
Obr99, WhiOO, LLC+02]. 

A finding of the 1999 Defense Science Board on Test and Evaluation [HAC+99] 
eoncluded that “Verification, Validation and Accreditation as presently praeticed with re¬ 
spect to M&S techniques in general is not sufficiently disciplined to inspire confidence in 
their use in the T&E process” [HAC+99]. [HAC+99] further proposed better eoordinated 
and more disciplined M&S software development proeesses supported by improved 
VV&A to address the following findings on M&S use in the test and evaluation: 

• M&S is effective and eost effeetive in many elearly defined applieations where both 
the expertise and data are available to support a common mission, 

• Earge-scale constructive force-on-foree eombat M&S tend to be the least useful 
when the inputs are largely unknown or uneertain, 

• The current Department M&S investment emphasizes model arehitectures, inter- 
faees, graphieal displays, and programming in lieu of coneeptual model develop¬ 
ment and basie data eollection, 

• No single, authoritative catalog or list of available M&S and quality attributes ex¬ 
ists, 

• Open air testing is not feasible for some weapon systems, ereating a dependency of 
some degree on M&S for evaluations of weapon systems, 

• M&S has potential in the test design and planning proeess, and the reliability, avail¬ 
ability and maintainability (RAM) areas, 

• Cost and benefits are diffieult to measure, up-front funding is often lacking, and 
there is a concern that the Department cannot maintain the talent necessary to im¬ 
prove M&S eapability [HAC+99]. 
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For all the potential benefits associated with M&S, the actual achievements fall 
short of predicted performance objectives, simulations lack widespread credibility to in s till 
user confidence, and the community still requires an acceptable method for determining the 
return on investment for M&S. [PHP+96] identified challenging, systemic, and recurring 
issues of credibility in the tools and serious doubts about the confidence of M&S results 
with the “underlying issues of the credibility of the tools being a major constraint” 
[PHP+96]. Only forty-five percent (20 of 44 systems) of the model summaries reviewed in 
a 1998 Department medical M&S catalog [Gau98] revealed any V&V activities. The in¬ 
ability to adequately quantify M&S fidelity and validity limits overall M&S utility. 

In a commentary of the current state of M&S [Whi99] suggests that VV&A has 
been an after-thought, a topic of discussion, but not properly implemented. These limita¬ 
tions acknowledges [PacOlb] make it difficult to establish confidence in M&S results, iden¬ 
tify the risks associated with decisions based upon M&S results, or determine the appropri¬ 
ate resource levels to increase confidence levels in the M&S. More importantly, these ex¬ 
amples are just instances of the many issues identified by [DOTOla] where the inadequate 
implementation of VV&A policies may result in the use of non-credible M&S to support 
critical Department program acquisition decisions. 

7. Secondary Credibility Contributing Factors 

In addition to the primary VV&A procedures, several other mitigating factors sur¬ 
faced often in the literature and appeared to have an important secondary affect on deci¬ 
sions to initiate or conduct a VV&A program. We call these secondary credibility- 
contributing factors, and these factors include reuse, documentation or lack of documenta¬ 
tion, risk management, cost of V&V, and the affect of V&V on a simulations return on in¬ 
vestment (ROI). Employed consistently, the secondary credibility-contributing factors 
support credibility, may reduce overall life cycle cost, and identify a return on investment. 
Unfortunately, the Department’s complex organizational dynamics, and complicated acqui¬ 
sition procedures [Wol02a, Wol02b] minimized the potential contributions of these factors. 
This list of credibility-contributing factors is not all-inclusive, however, we selected them 
for their potential influence on the development of software-intensive simulation systems. 
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a. 


Reuse 


Software reuse [IEE99a, IEE99b] is the “praetiee of using existing software 
eomponents^^ to develop new applieations” [DCC+93, KDM+94, IEE99a, IEE99b, Jaz02]. 
Software reuse has the potential to reduee development eosts, reduee time to market, im¬ 
prove simulation quality and support the V&V proeess, aeeording to [Som95] and [Pres97] 
who identify and explain the salient eonsiderations supporting suecessful software reuse. 

In 1991, the Department established the Software Reuse Initiative (SRI) to eollaboratively 
and eooperatively advanee the reuse strategy with other government ageneies, sueh as 
NASA, industry, and aeademia to make software reuse effeetive for the Department^® 
[RTC94]. Software reuse faeed many early ehallenges, whieh ineluded organizational ae- 
eeptanee, management support, legal issues, intelleetual property rights, liability eoneerns 
and ehanges to existing aequisition polieies [DCC+93, IEE99b]. 

Reuse evolved aeeording to [MooOl] and the eurrent foeus is now on “reus¬ 
ing knowledge itself, not just life-eyele artifaets sueh as speeifieations, eode, and test data” 
[MooOl]. Departmental M&S polieies also support reuse noting, “When possible, existing 
information systems shall be used or modified rather than ereating new systems” [DoD94]. 
[Pae02a] suggests that as the ability to automatieally generate eode from speeifieations im¬ 
proves, the foeus of reuse efforts should be in the earlier activities of the software life cy¬ 
cle, which precede coding, and smaller components may be easier to reuse. [TJOO, MooOl] 
reinforce this concept and propose domain engineering and analysis as the key to reusing 
knowledge, capturing common elements across the domain, to further additional domain 
development. 

In the M&S domain, explicit, well-documented conceptual models would be 
prime candidates for reuse, based on both definitions of the term reuse. Unfortunately, as 
[DoDOla] notes, past investments in M&S have not resulted in the Department’s capability 
to identify algorithms, data sets, models or simulations for reuse. Reuse of conceptual 
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Reusable software components can be executable programs, code segments, documentation, requirements, design and architectures, 
test data and test plans, or software tools. They may also be knowledge and information needed to develop, use, or maintain the compo¬ 
nent. [DCC+93] 

The DoD effort included the ARPA Software Technology for Adaptable, Reliable Systems (STARS) effort, the Army Software Reuse 
Initiative, the Navy Software Reuse Initiative, the Air Force Central Archive for Reusable Defense Software (CARDS), and the Defense 
Information Systems (DISA) Software Reuse Program (SRP). [RTC94] 
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models in the Department is inconsistent since, “Many Defense community simulations”, 
according to [Pac02a], “lack explicit descriptions of the conceptual models which are im¬ 
plemented in them” [Pac02a]. This is a central theme and key driver of this dissertation. 

Recent reuse techniques explored knowledge-based tutoring systems applied 
to software reuse [NLR99], computer-aided tools employing the relational hydrograph 
model and computer-aided software evolution system [Har99a], a software reuse reference 
model [RN99], the extension of existing reuse schemes with systemic software reuse meth¬ 
odologies and reuse success factors (e.g., planning, management support, process control, 
architecture, and domain focus) [RDK+99], and the employment of adapters to encapsu¬ 
late, and connect multi-use component interactions [RNJ99]. Other more abstract ap¬ 
proaches identified architecture reuse possibilities based on component-based reuse of a 
domain-specific software architecture [HBL97], architectural mismatches [GA095], and 
the architectural style approach [MG96] for classifying, storing, retrieving reusable archi¬ 
tectural design components. 

[HBL97] cites software component reuse [SLM91, LBG+96] and rapidly 
changing requirements [RL95, LBL+97] as the two major problems confronting large, 
complex software development, and suggests software evolution formalization as the best 
approach to understand the development and evolution process [BL94, DLB94, LG97]. A 
recent survey by [MET02] of large and small Europeans companies employing reuse, con¬ 
firmed these problems and found that approximately thirty-three percent of reuse projects 
in the survey population failed. Although the sampled companies belonged in different 
business domains, [MET02] found successful reuse projects shared good application com¬ 
monality, high process maturity levels, and use of standard procedural or object-oriented 
development methodologies. Successful reuse projects also shared the following attributes: 

• High commonality among applications, 

• Reuse is basically a technology transfer endeavor and requires management com¬ 
mitment, 

• The approach to reuse may be standard, but the deployment approach produced the 
best results when tailored to the context of the organization [MET02]. 

[RDB+00] suggests the current open source software movement, including 

Linux, is a logical extension of both the traditional reuse methodologies and other previous 
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open source standards including UNIX, Sendmail, Simple Mail Transfer Protocol (SMTP) 
for email, and the gee compiler. However, since reuse is not part of software engineering 
curriculums, [MET02] “forecast there will be a long delay before sensible advances are 
made” [MET02]. Progress has been noted in the reuse of communication protocols. When 
problems occur, the cause may be the mixing and matching of protocols, and using them 
for an unintended purpose (e.g., at the MAC layer rather than the session layer) [Mic03]. 

b. Documentation 

Software documentation is an important software artifact. Complexity of 
large software development projects drives the need for documentation to provide commu¬ 
nications between project members, customers, and management. [STS93a, STS93b, 
AS94] address documentation principles, methods, products, tools and management tech¬ 
niques supporting improved quality and productivity. 

Providing users with knowledge of M&S limitations and developing confi¬ 
dence in simulation results is a major Departmental M&S objective. [Sar98] supports 
model V&V documentation and a detailed basis-of-confidence document. [EKOO] pro¬ 
poses maintaining an assumption document to support the conceptual model development. 
This method also has potential to support a model basis of confidence document. 

However, documentation is usually a low priority for the software devel¬ 
oper. In addition, [Kil02] confirms that M&S capability^ ^ requirements, seldom under¬ 
stood by the user and rarely documented with useful detail beyond top-level users manuals 
continue to exist as major simulation development shortfalls. As a result, documentation 
for Departmental M&S systems is often incomplete, missing, outdated, or lacking detail 
[DoDOla]. 


Simulation capability descriptions should at a minimum include the purpose, modeled elements, environment, relationship, assump¬ 
tions, and limitations [Kil02a]. 
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c. Risk Management 

There are many widely used definitions for the eoncept of risk, including the 
IEEE’s definition [lEEOl]^^. In similar views, [BCC+00] defines risk as the probability of 
loss, while [HHG+02] views risk as the probability that a future event may or may not oc¬ 
cur, and the consequences. In this research we employ the risk management structure and 
definitions from the Risk Management Process Model [AndOl]. Risk is: 

A measure of the potential inability to achieve overall program objectives 
within defined program cost, schedule, and technical constraints and has two 
components: (1) the probability/likelihood of failing to achieve a particular 
outcome, and (2) the consequence/impacts of failing to achieve that outcome 
[AndOl], 

The Risk Management Process Model [AndOl] includes the following events, processes or 
procedures, varying with the phases of the system acquisition and system definition, and 
integrated into the program management function: risk events, risk management, risk plan¬ 
ning, risk assessment, risk handling, risk monitoring, and risk documentation. 

[HHG+02] also cites heuristic approaches to risk management including the 
probability of the future event occurring, the consequence of occurrence, and identifies 
guides, checklists and standard operating procedures to reduce or mitigate risk. [Boe91, 
Pfi98, Gal99, BCC+00, RPGOO, AndOl, HDK+02, HHG+02] identified software risk and 
risk management^^ approaches in software-intensive information systems including: guide¬ 
lines, checklists, and taxonomies. These methods also include the probabilistic techniques 
for ex post facto analysis or software reliability and formal statistical approaches detailed 
by [RPGOO]. 

In decision-making there are two basic types of risk: rejecting correct evi¬ 
dence or accepting incorrect evidence. In M&S software development the risks of accept¬ 
ing incorrect evidence are characterized in the [RPGOO] as either development or opera¬ 
tional risks. The Department’s W&A process, designed to reduce development risk by 
reducing errors and defects early in the development process, theoretically reduces the 
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Risk is the likelihood of an event, hazard, threat, or situation occurring and its undesirable consequences; a potential problem [lEEOl]. 
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Software risk management is a software engineering practice with processes, methods, and tools for managing risk in a project. It 
provides a disciplined environment for pro-active decision-making to assess continuously what can go wrong, determine important risks, 
and implement actions to deal with the risks [BCC+00]. 
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number of operational risks of ineorreet M&S outputs. However, [HHG+02] eonsiders 
software integration risk as the largest unknown “unknown” risk based on the eurrent soft¬ 
ware development paradigm of integrating varied and unstable software components such 
as database management systems, search engines and web browsers, which are constantly 
evolving to remain competitive in the market place. 

The risk of failure normally occurs in three ways and with different combi¬ 
nations of cost, performance, and schedule. These occurrences further decompose into the 
following risk areas of technical, supportability, programmatic, cost, and schedule risks. 
[BCC+00] identify several common risk factors including; management, ignoring risks, 
discipline, training, fallacy of easy solutions, work plans, schedule, delivery, constraints, 
customer, methods, and tool selection. Common risk-reduction methods, tools and proc¬ 
esses exist today to including project management tools such as PERT/CPM/Gantt charts 
and software estimation methods incorporated in software estimation tools [GAB97]. 

Multiple organizations in the private, government, and academia sectors 
continue to develop advanced risk management techniques, procedures, and automated 
tools, including the International Council on Systems Engineering (INCOSE) Risk Man¬ 
agement Working Group, the Project Management Institute (PMI) Risk Management Spe¬ 
cific Interest Group, the Camegie-Mellon Software Engineering Institute (SEI), Software 
Program Managers Network (SPM), the National Aeronautic and Space Agency (NASA), 
the Naval Postgraduate School (NPS), and the Defense Systems Management College 
(DSMC). The INCOSE Risk Management Working Group and the PMI Risk Management 
Specific Interest Group [HHG+02] collaborated on a joint project for a Universal Risk Pro¬ 
ject and developed a list and definition of universal risk areas^"^, applicable to any type of 
project or operation in the private and government sector^^. A universal risk is defined as 
“an event or condition that causes a deviation from the plan, with a reasonable chance of 
affecting the conduct of an analysis and that may occur in any project, operation or system. 
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The three major risk groups identified by consensus included: management risk area, external risk area, technology risk, [HHG+02] 
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This report used the INCOSE risk management process terms - planning, assessment, handling and monitoring. The PMI terms for 
the same steps are - risk management planning, risk identification, qualitative and quantitative risk analysis, risk response planning and 
risk monitoring and control [HHG+02]. 
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regardless of industry, organization or project/system type” [HHG+02]. In addition, 
[HHG+02] identified and defined speeifie risks under eaeh major group. 

The SEI risk management process approach [GAB97] consists of several 
functions preformed continuously over the project lifecycle; identification, analysis, plan¬ 
ning, tracking, control, and communication. [SPM95] explains an additional four areas of 
risk classified by the Software Program Managers Network: productivity, quality, timeli¬ 
ness, and user satisfaction. Recently, [HDK+02] presented potential improvements to De¬ 
partment risk management of the operation and implementation of information systems 
supporting vital defense functions by incorporating an approach called value focused think¬ 
ing. The NASA independent verification and validation (IV&V) methodology [ConOl] 
addresses a risk management approach, which is experiencing renewed emphasis following 
the less than successful results of recent NASA projects based on risks introduced into the 
projects by the implementation of the “better, faster, cheaper” management process 
[OCD+02]. 

Improving reliability of systems in the early software development stages 
when systems are especially prone to errors may be key to improving quality and the risk 
assessment process in software engineering. Recent research at the NFS by [LNOO, NJLOO, 
NLBOOa, NLBOOb, NLB+00, NogOO] focused on improving quality and reducing risk in 
the earliest phases of the software development project. In addition, [NLBOOa, NLBOOb] 
identified the inherent weakness of human dependencies and inconsistent application of the 
risk assessment process by different experts, and introduced a formal method with three 
metrics for requirements, personnel, and complexity supporting risk identification. 

New automated risk management tools continue to evolve. CASE tools 
such as the Computer Aided Prototyping System (CAPS) [LK88, Luq89, NLBOOa, and 
NLBOOb] support a risk assessment-based software evolution and prototyping approach. A 
risk analysis template [Whi99] consists of elements deemed to be important to the model. 
Applied to each element of acceptability for the model, it addresses the consequence of 
missing, poor or incorrect elements. To manage the risks introduced by these factors, 
[BK02a] introduced the credibility indicator trees method including fault tree analysis to 
support defined levels of V&V and credibility. Risk mitigation techniques directly affect 
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the entire simulation software life cyele, contribute to improved M&S credibility and sup¬ 
port the accreditation process. 

The objective of the Department’s W&A process is to reduce M&S risk 
[Kil02]. The verification process reduces the risk of unintended errors and the use of inap¬ 
propriate data, while the validation process improves confidence that the M&S outputs 
match the real world, and that the level of data accuracy is sufficient for the intended uses. 
Accreditation reduces the risk of selecting an inappropriate or unsuitable M&S for use. 
However, well-known V&V process constraints include the lack of time, budget limita¬ 
tions, an incomplete knowledge of reality, and complex M&S systems. Moreover, 
[HHG+02] contends that techniques are not the critical issue, but rather the lack of man¬ 
agement and engineering discipline ensures we continue to practice poor paradigms for 
system development. 

Mission-critical software systems may present high-risks to the overall sys¬ 
tem development. [CHK-i-90] identifies Mission Critical Computer Resources (MCCR) as 
the totality of computer hardware and software that is integral to a weapon system along 
with the associated personnel, documentation, supplies, and services. The DSMC devel¬ 
oped the MCCR process for four reasons: 

• Software for weapon systems is on the critical path of system development, 

• Software can experience significant development problems resulting in cost over¬ 
runs, 

• Modern weapon system performance depends on the quality of the computer re¬ 
sources and the system is only as good as the software, 

• Once a software development project falls behind, adding more resources (e.g., 
money, programmers) will not shorten the development times [CHK-l-90]. 

d. Cost of Verification and Validation 

There are concerns in the Department about the cost and lack of implemen¬ 
tation funding for conducting credible VV&A programs. Published VV&A costs range 
from five to eighteen percent of a projects funds cited by [GMS+96] with a mode between 
ten and twelve percent of the project cost. The cost of model validation noted by [Sar98] is 
usually non-trivial, particularly when confidence in the simulation results is important. 
Furthermore, developing absolute validity of a model over the entire domain for all poten- 
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tial uses contends [Sar98] is too costly, time consuming and probably impossible. Instead, 
[Sar98] suggests developing sufficient confidence by executing tests and evaluations as a 
means for determining a model valid for its intended application. 

The availability of documentation from previous W&A initiatives and un¬ 
ambiguous acceptance criteria also determines the cost of establishing M&S credibility 
[Mue97b]. Addressing the ever-increasing size of M&S, [MBZ99] conclude that the “in¬ 
creasing of scale and degree of complexity of M&S system[s] add [to] the difficulty and 
cost of VV&A process in life cycle of simulation” [MBZ99]. However, in a striking con¬ 
trast to [MBZ99] cost concept, [Whi99] contends that there is no additional cost for VV&A 
“since V&A activities are integral to normally accepted software development processes” 
[Whi99]. 


e. Return on Investment 

Major investments in Department M&S have not credibly established a re¬ 
turn on investment (ROI) cautions [BGK+00], and the challenge to implement credible and 
cost-effective V&V is growing. In many advanced technologies, providing confidence in 
performance to the specified level across the operating envelope depends to an unprece¬ 
dented degree on confidence in system M&S. For these reasons, [BGK+00] developed a 
business case framework to evaluate Departmental M&S investment opportunities. 

[Sta96, BGK+00] cites the failure to identify a return on investment (ROI) 
for M&S on the lack of PM staff knowledge, and the lack of Department institutional in¬ 
centives to build expensive models. In addition [BGK+00] compare and contrast case stud¬ 
ies of the commercial sector Boeing 777 aircraft and the Joint Strike Fighter M&S devel¬ 
opment within the Department M&S environment. Understanding the pressures that pro¬ 
gram managers are under to complete projects on schedule and within budget, [BGK+00] 
address several systemic impediments that preclude program managers from expending the 
funds, staff, or time to investigate the potential benefits of M&S. In order to improve this 
situation [BGK+00] develop a seven-step procedure to build a M&S business case, and re¬ 
iterate a recurring theme found in the body of research; recommending a disciplined ap¬ 
proach and methodology. 
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F. 


SUMMARY 


Chapter III introduced M&S credibility issues and identified concerns that decision¬ 
makers have with Department simulation results, followed by an overview of the Depart¬ 
ments VV&A processes, and several factors contributing to the implementation of, or 
shortfalls of the VV&A processes in the Department. Two major periods, the Growth and 
the Managed Eras characterize the initiatives, factors, and efforts to improve credibility in 
the Department’s simulation results. 

This chapter also addressed the credibility of M&S in the context of the procedures, 
policies and programs the Department established to improve decision-makers’ level of 
confidence in the results produced by the M&S. It is clear from the research that Depart¬ 
ment and the Service Components acted with due diligence to implement the policy, proce¬ 
dures, and guidelines to improve the quality of Department M&S. By the mid-1990s the 
Department established M&S management organizations, developed policies, implemented 
plans, allocated significant funding, and executed major new programs with the goals and 
objectives of improving Department M&S practices. However, not withstanding these 
considerable efforts, major concerns still exist about the credibility of the Department 
simulation process. There are several major reasons for this lack of credibility in Depart¬ 
ment M&S identified in the chapter. 

The new Department M&S management organizations, such as the DMSO, had to 
overcome Service parochialism, the unpredictability of the Department budget process, the 
vagaries of the Department acquisition process, and the life cycle management of over 
forty years of unplanned Department M&S growth. Then in the late 1990s significant 
funding and manpower resources originally programmed to improve Department M&S 
were committed to the Year 2000 problem resolution project. Research indicates three ma¬ 
jor categories of technical, cultural, and managerial challenges collectively hindered the 
development of improved Department M&S credibility. 

The research also identified an expanding role for M&S at the national policy and 
strategic level, and within the Department five major domains. A synthesized M&S Credi¬ 
bility Taxonomy established at Figure 3-1 identified interrelated factors affecting the De- 
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partment’s objectives for improved credibility, especially in the area of authoritative repre¬ 
sentations. There are many other factors involved in simulation credibility; however, these 
factors directly support the development of software-intensive simulations. 

The Department established the VY&A process in the mid-1990s as the primary 
method for establishing M&S credibility, prescribed regulatory requirements and provided 
best practices and techniques to the community. The Department continually cited a lack 
of credibility in M&S stemming from inadequate VV&A implementation negating deci¬ 
sion-makers’ confidence levels in the simulation results as a systemic issue needing major 
improvements, especially in the Department’s developmental and operational test commu¬ 
nities. Companion assessments of VV&A process implementation in the private sector and 
other public sectors indicated that the VV&A process failed to deliver the desired results. 

We identified and discussed VV&A principles illustrated in Figure 3-2, including 
the critical role that data plays in the VV&A process. However, the lack of authoritative 
data is a major deficiency in the Department’s attempt to improve M&S credibility. Activi¬ 
ties supporting credence in the M&S included model verification, conceptual model valida¬ 
tion, operational validity, data validity (Figure 3-3) and a model range of accuracy. Five 
key factors supported model credibility: capability, software accuracy, data accuracy, re¬ 
sults accuracy, and usability. The most definitive factor supporting credibility is to com¬ 
pare the model output with the actual or proposed system output data. We also identified 
and summarized common verification and validation techniques in Figure 3-4 employed by 
the Department, including M&S V&V techniques (e.g., informal, static, dynamic, statisti¬ 
cal, formal) identified in the [GMS+96 and RPGOO]. We noted that systemic issues identi¬ 
fied in by reports, studies, and assessments the early 1980s are still evident. 

The Department’s foundation for M&S credibility supporting confidence in simula¬ 
tion results is the M&S verification and validation process. Verified and validated concep¬ 
tual model support credible V&V processes. However, many consider V & V too expen¬ 
sive and time consuming. In this research we reviewed the V&V process in the context of 
supporting Department- and National-level decisions with improved credibility and confi¬ 
dence, and confirmed previous assessments indicating the quality of Department V&V 
practices are ad hoc and inconsistently applied. 
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The conceptual model of the mission space (CMMS or FDMS) is a critical compo¬ 
nent of a rigorous Department verification and validation program. The M&S community 
developed many conceptual model formats [RPGOO], however, there is no consensus on a 
standard conceptual model at this time. In this research we identified the underlying issues 
and propose a model supported by the software engineering process and a better under¬ 
standing of the complexity of the real-world fidelity definition and measurement. This re¬ 
search also expanded on the work accomplished by the SISO Fidelity Experimentation Im¬ 
plementation Study Group and their work on the Fidelity Conceptual Framework. 
[RGHOO]. 

During the research, several common themes continually emerged from the litera¬ 
ture: reuse, the lack of documentation, risk management, cost, and the identification of a 
quantifiable return on investment (ROI) from M&S. Reuse initiatives provided mostly dis¬ 
appointing results; however, “large-scale” reuse initiatives have the potential to improve 
this process. Documentation is a consistent problem in many software development pro¬ 
jects. In the M&S area in addition to the lack of software documentation, the lack of con¬ 
ceptual model documentation, coupled with the lack of documented V&V activities se¬ 
verely undermined the Department’s efforts to improve credibility. Risk management is a 
continuous activity applied during the entire life cycle of the M&S, where the early identi¬ 
fication and resolution of risks may be critical to improving overall M&S quality and 
credibility, appears uneven. 

The cost to accomplish V&V, constantly cited as being too high, may continue to 
grow as the M&S become larger and more complex. In addition, dentifying a quantifiable 
ROI has been an elusive goal for the Department, and as a result. Department acquisition 
program managers may be reluctant to invest the required resources in a tool, which re¬ 
quires significant resources today for questionable results tomorrow. 

The Department’s legacy M&S portfolio is a major study factor. The Department 
developed a significant and expensive portfolio of legacy M&S over the past forty years. 
Although an operational system has a de facto architecture, many of these M&S evolved in 
an ad hoc manner and have become large legacy systems with multiple stakeholders and 
expensive support infrastructures. A significant number of these systems lack conceptual 
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models and have an ad hoc V&V history. In some cases the developer created conceptual 
models after the fact, if at all, and adherence to disciplined M&S V&V process was a low 
priority. Department agencies may also have a tendency to use these systems for studies or 
tests for which they are ill suited, in order to show a return on investment. As a result the 
acceptability criteria and caveats developed by the accreditation process for these simula¬ 
tions may be of limited value or counter-productive to the original purpose and intent of the 
study or test objectives, and fail to establish credibility in the simulations or confidence in 
the simulation results 
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IV. HETEROGENEOUS SYSTEM REPRESENTATION ANOMALIES 


A. INTRODUCTION 

Improving the credibility of large-scale, legacy Department M&S requires an un¬ 
derstanding of the underlying systemic causes generating the pre-conditions for heteroge¬ 
neous system representation anomalies, especially in federation interoperability. Chapter 
IV reviews several heterogeneous challenges to Department M&S credibility. These di¬ 
verse challenges include syntactic and semantic heterogeneity, data heterogeneity, com¬ 
plexity, interoperability, technical and substantive interoperability, fidelity heterogeneity, 
and multiresolution modeling heterogeneity. 

B, ELUSIVE AUTHORITATIVE REPRESENTATIONS 

Authoritative representations have been consistently identified as major objectives 
in Department M&S, yet few of the [DoD95, DoDOla] objectives for authoritative repre¬ 
sentations have been achieved [MosOO, CraOla, CraOlb]. As a result of the long-standing 
unresolved systemic issues associated with Department M&S senior civilian and military 
decision-makers have significant reservations that Department M&S, as currently prac¬ 
ticed, can meet the serious future demands generated by national security requirements for 
M&S in the Department. In 1987, [Che87] placed boundaries on simulation results, later 
confirmed by [San97a] noting; 

Although simulations are useful tools, they are always approximations to re¬ 
ality, and, therefore, the credibility-the level of confidence that a decision¬ 
maker should have in their results—is open to question [Che87]. 

Current Department M&S users still require accurate, interoperable elements of the 
mission-space and the draft [DoDOla] retains authoritative representations as a Depart¬ 
ment-wide M&S objective. In addition [DoDOla] lists expanded requirements for credible, 
authoritative representations including U.S., allied, friendly, coalition, neutral and threat 
and paramilitary system representations for C4ISR, combat, combat support and combat 
service support systems and processes, supported by standard taxonomies and common ob¬ 
ject classes for systems representations by FY2004. Successful attainment of these inter¬ 
operability objectives by the Department and international M&S communities assumes the 
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acceptable resolution of many heterogeneity^^ issues, including abstractions, networks, 
federations and representations. 

Abstraction is an intrinsic human activity employed to reduce or filter the full 
eomplexity of the real world to manageable proportions by deleting unneeessary detail in 
order to provide data in context (e.g., information) supporting the deeision-making process 
[HJS97]. Abstraction mechanisms [Til98] selectively emphasize the level of detail, em¬ 
phasizing the details pertinent to the intended use, while hiding unneeded detail. [Sow88, 
Til98] discussed common abstraction mechanisms including: classification, aggregation, 
and generalization: 

• Classifieation is a form of abstraetion establishing an instance-of relationship be¬ 
tween the objeet type in the schema and its instance with an object defined as a set 
of instances with shared common characteristics, 

• Aggregation is a form of abstraetion establishing a part-of relationship between the 
eomponent objeets and the aggregate objeet, with relationships between objeets 
considered at a higher level aggregate object and specific details of the constituent 
objects hidden, 

• Generalization is a form of abstraction establishing an is-a relationship between the 
speeialized and generie object, when similar objects relate to a higher-level generic 
object and the constituent objects considered specializations of the generic object 
[Sow88, Til98]. 

C. DATA MODEL HETEROGENITY AND INTEROPERABILITY 


Data models establish the essential properties and well-defined relationships be¬ 
tween raw data in a system in a form, whieh supports efficient storage, timely retrieval of 
data, and provides the basis for tools and techniques to support data modeling [RagOl]. A 
data model captures the static and dynamic characteristics supporting data-related proc¬ 
esses, with statie properties defined in a database sehema and the dynamie characteristics 
developed into speeifications for reports, queries, and transactions. The sehema maintains 
the object type definitions, attributes, relationships and statie constraints of the data reposi¬ 
tory or database, an instance of the schema. 
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Heterogeneous is defined as: consisting of or involving dissimilar elements or parts. Heterogeneous networks are a collection of simu¬ 
lations with partially consistent behaviors and/or partially correlated databases. Examples include simulators of different fidelity, mixed 
virtual and live simulations, and mixes of virtual and constructive simulations [DoD98]. 
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Static characteristics include objects, attributes, and relationships among objects [Til98]. 
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Dynamic characteristics include operations on objects, operation properties, and relationships among operations [Til98]. 


80 



The most significant record-based logical data models includes the hierarchical data 
model, organized into simple tree structures; the network data model, a superset of the hi¬ 
erarchical data model, without the requirement for tree structures; and the relational data 
model, based on a mathematical foundation of a relation (e.g., a set of n-tuples) organized 
as a related collection of tables [01183]. These classic highly machine-oriented, and record- 
oriented data models, or syntactic data models [Til98], are well suited to the computer en¬ 
vironment and organized for optimal efficiency supporting storage and retrieval operations, 
however these models lack semantic relativism^^ and abstraction mechanisms required 
for dealing with complexity in large-scale systems [Til98]. 

[Til98] describes a second category, semantic data models, which combine basic 
knowledge representation techniques with database technology, and represent a transition 
from the basic record-oriented approach towards a model supporting more human-oriented 
semantic constructs. While traditional data models store data, and semantic models shift 
towards a more human knowledge model, [Til98] suggests a third method, the conceptual 
modeling^^^ activity geared more towards the domain-level knowledge and program under¬ 
standing, with a focus on the end-user. Program understanding supports development of a 
fidelity-based product line conceptual model referent domain-architecture, since large leg¬ 
acy systems represent substantial corporate knowledge of an organization’s business rules, 
requirements, and design decisions. [Til98] further suggests that all three techniques may 
be more suitable than one method if there are different categories of artifacts (e.g., data, 
knowledge, information), requiring a relational model physical storage of data, a concep¬ 
tual model for representing domain-level knowledge, or a semantic model for interactive 
discovery. 


Semantic relativism is the ability to view the elements and concepts representing a modeled system from different perspectives based 
on the application [Til98]. 

The most common abstraction mechanisms are classification, a form of abstraction in which an object is defined as a set of instances; 
aggregation; and generalization, a form of abstraction in which similar objects are related to a higher level generic object [Sow88] 
Conceptual modeling (e.g., conceptual schemata) in this context is the activity of formally describing aspects of some information 
space for the purpose of understanding and communicating, with an emphasis on the knowledge organization used by humans, rather 
than the data organization used by machines [Til98]. 
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D. HETEROGENEOUS SYNTACTIC DATA MODELS 


[Wie99] submits the objective of interoperation is to increase the value of informa¬ 
tion when accessing, relating, and combining information from multiple sources. In addi¬ 
tion, [Wie90, Wie93, Wie96a] acknowledged that data comes from many diverse and het¬ 
erogeneous sources and further suggested that joining heterogeneous data is necessary to 
generate information or compose large-scale software applications [MBS+99]. 

[Wie90, Wie93, Wie96a] also addressed several contributing factors for differences 
among systems in their research on database schema integration including the different per¬ 
spectives of the developers and users; the use of equivalent constructs (e.g., different uses 
of the same constructs to create the same representation); and incompatible design specifi¬ 
cations. Focusing primarily on database constructs and schema research, [WW90, Wie93, 
Wie96a, WieOOa, WieOOb] developed eight viewpoints for differences among systems, and 
[You02c] added a ninth viewpoint of heterogeneity germane to this research: 

• Heterogeneity of Hardware and Software systems, 

• Heterogeneity of Organizational Models including network, hierarchical, relational, 
universal database models and object-structured data, 

• Heterogeneity in Representation since it causes anomalies at the limits or bounda¬ 
ries (e.g. 5-digit zip code versus 9-digit zip code), 

• Heterogeneity of Scope with different databases capturing different views of the 
same real-world object (e.g., employees paid versus employees available), 

• Heterogeneity of Level of Abstraction manifested by different aggregation schemes 
(e.g., personal income versus family income), implying the existence of an aggrega¬ 
tion hierarchy, 

• Heterogeneity of Meaning addressing assignment of attribute values to real-world 
attributes (e.g., postal codes versus town names), 

• Heterogeneity of Temporal Validity caused when values have, often implicitly, dif¬ 
ferent temporal ranges (e.g., monthly budget versus weekly production), which may 
not support aggregation [Wie93], 

• Semantic Heterogeneity occurs because the meaning of words depends on the con¬ 
text, and data sources develop within their own context. Types of Semantic Het¬ 
erogeneity include synonyms, spelling differences, and the use of identically spelled 
words to refer to different objects [WieOOa, WieOOb], 

• Heterogeneity of Structure causing a variation in the structure of the information ar¬ 
rangement between systems employing the same organizational model (e.g., a plane 
modeled as an attribute in one system and an object in another [You02c]. 
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In supporting a decision-maker, [Wie02] suggests access to a variety of methods 
and heterogeneous information sources for predicting the future including extrapolation, 
interpolation, projections, published projections, spreadsheets, discrete simulations, and 
continuously executing simulations (e.g., weather predictions). Many methods are compu¬ 
tation intensive, and except for desktop models such as spreadsheets, most simulations are 
too complex, costly, or require too much manpower or time to meet decision-makers re¬ 
quirements for information supporting analysis of alternate future courses of action. In ad¬ 
dition, distributed simulations employing HLA, DIS, or ALSP protocols rarely interface 
with other external types of information systems such as databases, although simulation 
systems may include internal databases or files to retain past, and near-current data for ta¬ 
ble lookups and scenario support to the simulation. Relational databases support the Struc¬ 
tured Query Language (SQL) for database access; however, SQL currently has no native 
capability to communicate with a simulation or a federation. 

[WieOOa, WieOOb] notes that semantic heterogeneity causes failure to find the de¬ 
sired object, and affects the level of precision in selection, aggregation, and comparison 
when integrating information. Furthermore, [WieOOa, WieOOb] believes a global solution 
to semantic mismatch is unfeasible due to the number of participants, terminology changes, 
and scope, but notes that precise, finely differentiated terms and abbreviations may be suit¬ 
able for domain efficiency. For instance, [WieOOa, WieOOb] opines that consistency is lo¬ 
cal (e.g., precision for an on-line commerce requires a consistent structure and a consistent 
terminology for the same set of objects). 

[WW90, Wie93, and MW02] suggest that knowledge of the enterprise operation 
supports the development of the best possible information from diverse sources, and also 
note that since heterogeneity is never completely resolved, uncertainty heightens in the de¬ 
cision-making process. In the same vein, [WieOOa] suggests that many small-scale efforts 
provide superior results over major standardization efforts, and proposes that precision and 
relevance attributes will be more valuable than completeness and recall attributes. Preci¬ 
sion, with its many components (e.g., rule-based matching, metadata, quality attributes, and 
processing models) is an important aspect of information, and will become increasingly 
important [WieOOa]. 
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[Wie99] also notes that interface standards are critical for building and maintaining 
multi-layer systems, citing the role of SQL for structured tables, CORBA and DCOM for 
middleware; and XML when data cannot be structured well. [Wie99 and DM03] further 
suggest using XML for specific domains with domain-specific type descriptions or sche¬ 
mas, to help in matching the meaning of shared information. 

As a possible solution to the heterogeneity challenge, [WW90, Wie97a, Wie99, 
WieOOa, WieOOb] proposes a two-dimensioned mediator architecture partitioning resources 
and services to resolve heterogeneity issues with three horizontal layers (e.g., client appli¬ 
cations, intermediate service modules, and base servers), and vertically into any domains, 
with a limited number of supporting servers. Noting that any implementation for a system 
presenting past and future information requires the cooperation of specialists in distinct 
fields and an openness to learn new methods and techniques, [Wie96a, Wie97a, Wie02] 
identified a three layered hierarchy, with an intelligent middleware mediator layer linking 
the heterogeneous information sources at the bottom with the application programs at the 
top to achieve an intelligent integration of information. The mediator performs the follow¬ 
ing tasks: 

• Accessing and retrieving relevant data from multiple heterogeneous information 
sources, 

• Abstracting and transforming retrieved data into a common representation and se¬ 
mantics, 

• Integrating the homogenized data according to matching keys, 

• Reducing the integrated data by abstraction to increase the information density in 
the transmitted result [Wie97a]. 

[Wie97a, WJ98, Wie99, Wie02] demonstrated a capability for supporting integrated 
predictive capabilities from spreadsheets and other simulation tools into information sys¬ 
tems, employing an interface language, SimQL, combined with wrappers. SimQL provides 
a schema, describing the accessible contents, and a query language to access the informa¬ 
tion resources, similar to SQL capabilities [Wie99] and has four major components: 

• A compiler for the SimQL language, generating code to access wrapped resources, 

• A repository containing the schemas for the wrapped resources with input and out¬ 
put parameters identified, 

• A wrapper generation tool to bring simulations [WJ98], spreadsheets, and web re¬ 
sources into compliance. 
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• The source forecasting tools, spreadsheets, discrete simulations, and web sources 

[Wie99]. 

Other methods for composing components from heterogeneous, autonomous, and distrib¬ 
uted information sources include CPAM [MBS+99], an image indexing and retrieval algo¬ 
rithm, WBIIS, providing a search capability for a large image database [WWF-i-98], and the 
Object-Oriented Method for Interoperability (OOMI) [You02c]. 

The [You02c] solution is a federated interoperability object model (FIOM) and an 
automatic translator generator. [YBG+01, You02c] developed the FIOM, an interoperabil¬ 
ity object model defined for a specific federation of systems, and an interoperability object 
model for interoperability (OOMI) with a corresponding structure. [YBG+OI, You02c] 
also envisions the OOMI as an extension of the contemporary object model with a class 
structure property and extended attribute and operation properties to capture the different 
representations. The [YBG+01, You02c] translator function is a software wrapper, com¬ 
ponent of a message-based architecture, or as part of the data store, with an XML-based 
message translation currently under study for implementing the proposed model. Where 
[You02c] provides a necessary near-term, interim solution for interoperability via the Ob¬ 
ject-Oriented Method for Interoperability (OOMI) translators, this research addresses the 
root cause issues that effect simulation substantive interoperability in high-risk, mission 
critical simulation systems, and addresses the issue at the first level of abstraction, the ar¬ 
chitecture. 

Research continues reveal mew methods to fuse information from heterogeneous 
resources including databases, text, semi-structured information bases [WW90, Wie97a] 
involving intrinsic semantic differences [Wie03], supported by complementing ontology 
research [Gua98, MW02]. Commercial software vendors also continue to expand basic 
database systems, adding communication, data mining, and analysis functionality to the 
basic database capabilities supporting the decision-makers need to plan and schedule future 
actions, and assess the effects of alternate future courses of action [Wie99]. Data reposi¬ 
tory holdings of past and near-current data provide only limited background information 

The semantics of an information source is captured by its ontology, the collection of domain terms and 
their relationships used in the discourse [MW02]. 
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for assessing alternate future courses of action, while decision-makers require a spectrum 
of heterogeneous information including past, present, and likely future situations [Wie99, 
DoDOla, Wie02, Wie03]. Simulations, coupled with other heterogeneous information re¬ 
sources, may provide a potential new information source to support a decision-makers 
analysis of alternate future courses of action. 

E. HETEROGENEOUS CONCEPTUAL MODELS 

The Department addressed many M&S problems in the 1990’s to improve overall 
quality, including technical and substantive interoperability. Although there has been pro¬ 
gress with the technical interoperability problem since the mid-1990s, many approaches 
failed to improve the credibility of authoritative representations and reduce the federation’s 
interoperability problems caused by substantive interoperability anomalies. This results 
from simulation developers abstracting real-world objects in their model representations 
from different perspectives, requirements, constraints, and objectives. 

Over time developers modeled many of the same objects in different ways, poten¬ 
tially causing problems when they interact in a federation. [DSB+99, HYOla, HYOlb, 
YPHOl] identified systemic substantive interoperability issues with legacy simulation 
software systems and the inconsistent representation of the same entity in multiple systems. 
There is a very close correlation on the root causes between the technical and substantive 
interoperability issues cited in the simulation domain, and the previously cited work of 
[Wie93, KinOl, You02c] on developing information from heterogeneous data sources. 

This research revealed significant progress in the area of technical interoperability, 
including research into multilevel simulation of discrete network models [Ham96], distrib¬ 
uted simulations [Ham96, HNP97], and the Department-sponsored development of the 
HLA. These advances supported the ability of federates to physically connect and ex¬ 
change data employing a federation object model, the use of common standards, and coor¬ 
dinated data structures. The HLA infrastructure supports hardware and software compati¬ 
bility, standards compatibility, time management, security, and coordinated use of the RTI. 

However, federation interoperation currently requires significant manpower and co¬ 
ordination, and often encounters technical and substantive interoperability issues during 
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operation. Substantive interoperability has been a systemie issue since the Department’s 
earliest experiences with networking simulations, SIMNET, and persisted through the evo¬ 
lution of the ALSP, DIS and HLA protocols. Today, substantive interoperability, including 
entity interoperability issues remains a major challenge for achieving truly distributed 
simulation interoperability [DoD03a]. Entity interoperability includes a logical interaction 
between entities modeled in different simulations, temporal resolution, spatial resolution, 
and a coherent relationship between the components of the physical environment (e.g., 
ground, ocean, atmosphere, weather, and the electro-magnetic spectrum) to support the in¬ 
teroperability context [YPHOl]. Entity interoperability also encompasses the level of rep- 
resentation , attributes and behavior including: 

• The required federation entities are available at the necessary level of detail, 

• The federation entities fit together, 

• The required federation entities meet the critical characteristics of the real-world en¬ 
tities, 

• The required federation entities possess the salient attributes supporting the federa¬ 
tion’s intended use, 

• The required federation entities accurately model the needed behavior, 

• The required federation entities work together logically, 

• The required federation entities employ consistent algorithms to compute effects 
[YPHOl, HYOla]. 

The RTI time management services support federation synchronization of individ¬ 
ual federate time and attempts to manage federation temporal resolution assuming: 

• Each federate computes change in the entity states at some resolution of simulation 
time, 

• Incremental state changes occur often enough to support logical interaction between 
entities, and allow the exchange of data to occur [YPHOl, HYO la]. 

The spatial resolution shared among the federates must also support the intended 
use, although the federates are not required to compute the or store the entity spatial data in 
the same reference frame assuming: 

• Each federate computes the location of entities on a specific geospatial reference 
system. 


Level of detail - Entities do not require the same resolution, as long as the interactions between entities meet the intended use of the 
federation [YPHOl]. 
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• The computed and shared entity locations allow different federates to adequately 
and consistently meet required federation geospatial and other environment objec¬ 
tives [YPHOla]. 

The environment is a critical factor for achieving substantive interoperability, with 
each federate developing a sufficiently consistent correlated view of the environment, in¬ 
cluding both visual (e.g., tanks, planes, buildings, water) and non-visual spectrums (e.g., 
radio frequency (RF), infrared radiation (IR), photons (lasers)). Early efforts enforced ho¬ 
mogeneity by developing a mapping between the different simulation environmental repre¬ 
sentations, while today there are integrated data set solutions including a standard syntax 
and semantics with the SEDRIS data reference model providing a standard interface speci¬ 
fication, data dictionary, and data coding standards [YPHOl]. Research continues in repre¬ 
senting the effects of the environment on systems and human behavior (e.g., the detection 
capability of an IR or RE sensor in a perturbed environment), and conversely on the impact 
of systems and humans on the environment (e.g., the impact of chemical or radiation weap¬ 
ons in a city). 

A few analogous examples of the current challenges presented by the lack of tech¬ 
nical and substantive interoperability provided below illustrate, compare, and contrast the 
concepts of technical and substantive interoperability: 

• You dial a number for an important business call, but accidentally reach a computer 
modem and hear the sound of a modem trying to synchronize with a sending mo¬ 
dem. The communication network performed as designed. The communication 
protocols worked up to a point, however, you received no message. This represents 
a passing grade for technical interoperability, but a lack of substantive interopera¬ 
bility. The result: time lost redialing, maybe lost business since the message did not 
get through on time. 

• In another example, arriving home after work and just as you prepare to relax for a 
late dinner, the phone rings and you are greeted by a telemarketer or worse yet, no 
response, indicating an autodial technique advancing ahead of the telemarketer’s 
verbal proposal. In this case you hung up the receiver even though the communica¬ 
tion network and protocols worked exactly as designed. This case illustrates suc¬ 
cessful technical interoperability and a lack of substantive interoperability, how¬ 
ever, since the telemarketer‘s message failed to go through. The result: no message 
delivered and no sale for the telemarketer. 

• In an HEA federation scenario the publisher submits a number 0.506127 for a state 
variable in a specific instance, and the federation subscription process apparently 
works. Initial analysis suggests the results are normal. Subsequently following a 
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major live test event sueh as a flight test, post-test simulations runs exeeute, but er¬ 
rors in proeess truneate the value to three digits, e.g., 0.506 from the 0.506127 
number needed for the required preeision. Multiple runs of the simulation exeeute, 
but the results appear ineonsistent. Several faetors in the host proeessing systems 
ineluding translation, level of preeision, or even perturbations due to ehaos theory'®"^ 
may be the root eause. The result: substantive interoperability anomalies in an 
HLA federation, ineonsistent results, laek of eredibility in the simulation and loss of 
eonfidenee by the user. 

Substantive interoperability anomalies oeeur when one of the following types of 
representational anomalies manifests themselves: 1) event phase anomalies, 2) event order¬ 
ing anomalies, 3) state error anomalies or registration anomalies[HYOla, YPHOl] and 
ereate intolerable representational anomalies. Representational anomalies beyond aeeept- 
able toleranees for the federation’s intended purpose ean manifest themselves from an inva¬ 
lid federate and/or from interaetions between valid federates. 

Tests may identify the source conditions of possible interoperability-related anoma¬ 
lies: including functional compositions and manifold representations [HYOla]. Functional 
compositions occur when the computation of one or more object states in one federate de¬ 
pend upon the data provided by another federate and violates one or more of the following 
interoperability criteria adversely affecting M&S quality: 

• Dependency representation, 

• Representational accuracy, 

• Range consistency, 

• Sensitivity consistency, 

• Temporal representation, 

• Internal sensitivity, 

• Error consistency, 

• Stochastic consistency [HYOla]. 

Manifold representations, normally exhibited under very specific conditions, occur 
when two or more federates: 1) represent the same behavior or state, and 2) interact either 
directly or indirectly. Both parallel manifold representations and sequential manifold rep- 
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This phenomenon is common to chaos theory, and is also known as a sensitive dependence on initial conditions, first identified by 
Edward Lorenz, a meteorologist, in 1960. 

Event phase anomalies exhibit a timing or phase error. Event ordering anomalies occur when the simulated object produces the same 
events as the simuland under identical conditions, but in a different order. State error anomalies are identified by a difference between 
the state a simulated object assumes and the state that object’s referent exhibits under identical conditions and the difference is beyond 
tolerance levels [HYOla]. 
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resentations may cause interoperability problems, including aggregation or disaggregation 
issues if they fail to meet manifold representation interoperability eriteria including: state 
correspondence, abstraction transform, or state continuity. The simulation eoneeptual 
model provides the neeessary supporting information to support these tests [PaeOla], 
while the Federation Development and Execution Processes (FEDEP) [GYE03] inelude 
conceptual model development proeedures and guidelines for building a federation, sup¬ 
porting eredible verification and validation [DSB-l-99]. 

The level to whieh heterogeneous system representation anomalies affeet the credi¬ 
bility of a federation is derived from the representational aeeeptability criteria, based on the 
federation’s purpose and the capabilities the federation or federate must meet to support 
that purpose [YPHOl]. T\iQ DoD Modeling and Simulation Verification, Validation Ac¬ 
creditation Instructions [DoD96] supports eredible M&S development and ineludes ac¬ 
ceptability criteria^'^^, linking the roles and responsibilities of federation life-cycle devel¬ 
opment and operation with improved decision-makers confidenee in the results when: 

• Developers design and implement the software funetionality to satisfy the criteria, 

• V&V agents evaluate federation capabilities against the criteria, 

• Acereditation agents make accreditation reeommendations based upon the eriteria 

[YPHOl], 

Department decision-makers accredit the use of the federation for a specific purpose 
based on confidenee that the aeeeptability criteria were credibly established and imple¬ 
mented. A contributing factor to the substantive interoperability problem has been an in¬ 
consistent implementation of the Department’s verification, validation, and accreditation 
(VV&A) policy identified in Chapter III. The Department’s W&A processes depend on 
consistent, credible eoneeptual models. However, Department’s eoneeptual model imple¬ 
mentation is inconsistent, and the theoretical debate over conceptual model formats, meth¬ 
ods, uses and development processes eontinue unabated. In addition data to validate De¬ 
partment’s M&S remains elusive, expensive and difficult to acquire, adding to the chal¬ 
lenges of eonducting creditable VY&A. 


The simulation conceptual model is the developer’s translation of modeling requirements into a detailed design framework, from 

which the software, hardware, networks and systems/equipment that make up the M&S can be built [HYOla]. 
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Acceptability criteria should be necessary for the purpose, specific, measurable and complete [YPHOl] 
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Substantive interoperability issues also exist due to the level of software engineer¬ 
ing discipline and the general lack of process maturity exhibited by Department M&S 
software developers (see Chapter VII). The Department initiated major new large-scale 
M&S development efforts, JSIMS, JWARS, and JMASS, to replace many of the aging leg¬ 
acy systems and decrease the issues with substantive interoperability. However, all three 
major systems are well behind schedule, significantly over-cost, and failed to demonstrate 
they meet the original requirements. The development and implementation of software ar¬ 
chitectures, including product lines, in the Department while still very immature as it mir¬ 
rors progress made in the private sector, has the potential to improve the Department M&S 
substantive interoperability problem in the future. Compounding the VV&A deficiencies is 
the use of over-subscribed terms for M&S quality attributes such as fidelity. 

F, FIDELITY HETEROGENITY 

Chapter VI discusses M&S quality attributes, including fidelity, and Table 6-2 pro¬ 
vides the current Department context for M&S fidelity. However, continued concern about 
M&S fidelity within the SISO led to establishment of two fidelity study groups in the late- 
1990s with charters to produce a fidelity conceptual framework for M&S fidelity and a 
glossary of fidelity related terms [Gro99]. The Simulation Standards Interoperability Or¬ 
ganization chartered the Fidelity Definition and Metrics Implementation Study Group 
(ISG-FDM) [Gro99] and a follow-on effort, the Fidelity Experimentation ISG (ISG-FEX) 
[RGHOO] to address the issue of fidelity. These efforts produced collaborative integrated 
project reports addressing appropriate M&S fidelity definitions, a fidelity conceptual 
framework, formulas and concepts for the SISO community, and recommendations for fu¬ 
ture DMSO M&S efforts. However, fidelity, the subject of many scientific papers, studies, 
analyses, and reports, still lacks an acceptable, quantifiable methodology. 

The [Gro99, RGHOO] reports, a major SISO collaborative effort by over one hun¬ 
dred researchers, provided a summary of workable contextual frameworks within which 
M&S fidelity is considered, with an explicit indication of how such frameworks would re¬ 
late to the larger theoretical context of modeling and M&S theory, including initial identifi¬ 
cation of methods, approaches, and metrics for usefully defining, estimating, and measur- 
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ing aspects of M&S fidelity. Fidelity of a model or simulation defines the terms of the 
relevant referent and the capabilities of the model or simulation. Fidelity also describes the 
essential characteristics of the model or simulation relative to its referent. 

[Gro99, RGHOO] also identified the dimensions and attributes of M&S fidelity 
based on the characteristics of reality, addressed by entities, factors and relationships 
within the simulation’s enumeration of the problem’s critical aspects. [Gro99, RGHOO] 
further defined the scope of involved entities and the identifiable depths of entities. 

[Gro99] identified three dimensions of simulation fidelity, which include a number of the 
possibilities, or a percentage of factors 

The first dimension of simulation fidelity [Gro99] identified is enumeration of the 
involved entities, addressed in both scope^°^ and depth'°^. In the second dimension of 
simulation fidelity [Gro99] included the identification of involved factors relating the 

processes, which influence, impact, or describe entity states and behavior. Specification of 
the significant relationships among entities is the third dimension of simulation fidelity 
forwarded by [Gro99] to explicitly articulate the assumptions about dependencies or inde¬ 
pendence among entities. [Gro99] also describes dependent relationships not explicitly 
identified as “otherwise”^ 

Fidelity presents significant challenges for accurately addressing the size, scope, 
and complexity of the real world and our current understanding of the real world preclude 
its use as a practical measure. Since the real world is not a good foundation to measure fi¬ 
delity, research efforts have identified the need for common referent, a commonly under¬ 
stood standard [Gro99, RGHOO]. This raises the point of how to assess the fidelity of a 
simulation against only those aspects of the referent needed to support the simulation; oth- 
erwise, if the simulation represented all aspects of the simuland it would in fact be the 
simuland [Gro99, RGHOO]. 


Scope addresses the spectrum of entities represented by the simulation [Gro99]. 
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Depth is the level at which entities in a simulation can be individually identified [Gro99] 

Factors may include components, materials, internal parameters, algorithms, and parameters related to measures of performance 
(MOPS)/effectiveness (MOE)/ merit (MOM) [Gro99]. 

The “otherwise” category described by [Gro99] as a description for all dependent relationships that cannot be described more pre¬ 
cisely. 
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A simuland is the system being simulated by a simulation [DoD98]. 
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The second obstacle [Gro99, RGHOO] identified is the specification of the simula¬ 
tion. Defined simulation fidelity has the potential to act as a metric for describing how well 
the behavior of the unique aspects of the simulation matches selected parts of the simuland. 
[Gro99, RGHOO] suggests the simulation also has many other characteristics describing the 
nature, behavior and character of the simulation, independent of the simuland, requiring a 
complex, multidimensional set of measures. The multidimensional set of measures in¬ 
clude, but is not limited to, the simulation’s intrinsic and extrinsic quality; development 
costs; resources needed to develop scenarios and execute the simulation; the extent that de¬ 
composition, aggregation and interfaces must be described; and the intended computation 
environment [Gro99, RGHOO]. 

Simulation fidelity attributes address the quality of the parameters within the di¬ 
mensions of M&S fidelity and include the following characteristics: factor order, accuracy, 
precision, timeliness, consistency, repeatability, and possible error sources [Gro99, 
RGHOO]. Generally, within simulations the higher the factor order[Gro99, RGHOO] 
suggests the greater the fidelity. However, since factors may be high order, low order, or 
some other order within the simulation, [Gro99, RGHOO] suggests terms of factors provide 
better definition of simulation fidelity in, in lieu of using the concept of aggregation. Other 
simulation fidelity attributes include consistency^repeatability^'^, and people issues"^. 

A closely related concept including aspects of abstraction and aggregation is multiresolu¬ 
tion modeling. 
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The quality of parameters within a simulation start with the order of the parameter description of the factor in the simulation with a 
zero order parameter description being a constant value, a first order parameter description would vary in one way according to some 
statistical distribution, a second order parameter description would very in two ways, etc [Gro99, RGHOO] 
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Consistency addresses whether simulation results are biased and stable in terms of the dispersion of results created by the simulation 
processes [Gro99, RGHOO]. 

Repeatability means that a simulation should produce the same results given the same stimuli [Gro99, RGHOO]. 

People issues, often ignored, is a special fidelity concern since as players, operators or analysts can impact accuracy, precision, con¬ 
sistency, and repeatability in simulation results [Gro99, RGHOO]. 
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G. MULTIRESOLUTION MODELING (MRM) HETEROGENITY 

[Zei91, Zei92a, Zei92b, Dav92a, Edd92,] introduced concepts supporting variable- 
resolution modeling (VRM), a precursor to multiresolution methodology (MRM) . These 
early citations provided definitions, introduced basic concepts, identified types of VRM, 
provided examples, illustrated the importance of VRM for integrating new models or redes¬ 
igning existing models, including potential cost savings [Sil92]. [Dav92a, Dav93] also dis¬ 
cussed three classes of variable resolution they termed: selected viewing, alternative sub¬ 
models (or model families), and integrated hierarchical variable resolution (IHVR). The 
IHVR concept forwarded [Dav92a, Dav93] by defines critical processes, hierarchically 
composed of subordinate processes. 

The study effort also built on previous MRM research by [Ham96, HNP97, Dav98, 
DB99, SHBOO, and DB03], which proved invaluable for establishing a foundation support¬ 
ing the introduction of software architecture-based M&S product lines. [Dav98] notes that 
MRM is “still a frontier subject in modeling and simulation” [Dav98], with more work 
needed in diverse areas including fundamental theory, computational tools, and visualiza¬ 
tion techniques. Important theoretical predecessor works include object-oriented modeling 
and discrete-event simulation concepts by [Zei91], modular hierarchical model representa¬ 
tions by [Zei92a], a review of stochastic versus deterministic methodologies in combat 
simulations by [LucOO], and the development of methodologies for structured families with 
multiple levels of resolution [Zei92b]. 

Furthermore, [DB99] recognized the limitations of programming and operations re¬ 
search and identified a need for in-depth research “that leads to sound theories and designs 
well before coding even begins,” [DB99] and “should reflect sound principles of software 
engineering, including the ability to design in an object-oriented modeling framework” 
[Dav98, DB99]. In lieu of combining several different models at different levels of resolu¬ 
tion into a federation or other form of distributed simulation operation, [Ham96, HNP97] 


Some of the many terms that MRM issues arise under are model abstraction theory, problem decomposition, variable-resolution 
modeling, variable-fidelity modeling, hierarchical modeling, aggregation and disaggregation, chunking, and building of lumped, reduced, 
or simplified models [Dav98]. 

[Zei92b] suggests that resolution is the degree to which objects and processes in the real world are resolvable in the model, and fur¬ 
thermore, the resolution level is synonymous with the level of detail [Zei92b]. 
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discuss an alternative strategy of developing and operating at multiple levels of resolution. 
[DBW87, BM92, Dav92a, DH92a, DH92b, Dav93, Ham96, DC97, HNP97, Dav98, DB99, 
MDBOOA, MDBOOb, DavOO, DBMOO, BDMOO, and AdeOl] developed a significant body 
of knowledge on multiresolution modeling and multi-resolution multi-perspective model¬ 
ing (MRMPM). 

[DB99] further defines multiresolution modeling'[DH92a] noted that achieving 
interoperability in diverse, independently developed M&S with different, but overlapping 
resolutions, based on different assumptions, limitations, and perspectives is a major chal¬ 
lenge in contemporary M&S. [Dav92a, Dav93 and DBM99] in Figure 4-1 illustrate that 
the term resolution represents many different concepts and dimensions. 

However, [DavOO] noted MRM may not be sufficient for applieations requiring dif¬ 
ferent abstraetions or perspectives, that vary by eonception of the system or use of vari¬ 
ables, and requires a new model construct for both multiple resolution and multiple per- 
speetives (MRMPM). In [MDBOOa] models and families of models employed hierarchical 
trees to remain mutually consistent across multiple resolution and perspectives. [BDMOO] 
employed an exploratory analysis teehnique using low-resolution models at the macro level 
of analysis and high-resolution models to support validation efforts. 

Adding to the existing body of knowledge, [DavOO, DBMOO, BDMOO, and Dav93, 
and HOB95] eompiled a signifieant body of work in multi-resolution, multi-perspective 
modeling (MRMPM) complementing the existing aggregation / disaggregation research. 
Taking a hybrid top-down /bottom-up approach to validation, [Dav95a] proposed a family 
of models optimally designed top-down and bottom-up concurrently, noting that the more 
detailed models may only have selected details and may lack the information required to 
develop valid aggregate models. [HNP97] suggest the development of a resolution hierar¬ 
chy supporting the ability to control the bounds of inputs. [Dav95a] further indicates ag¬ 
gregated models may be able to establish the context and boundary conditions for events 
required in models that are more detailed. [Dav95b] applied the combat ratio 3:1 rule to 


Building a single model, a family of models, or both to describe the same phenomena at different levels of resolution [DB99]. 
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In combat operations the ratio of attackers to defenders for a successful offensive operation [Dav95b]. 
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the complexities of aggregation and disaggregation and detailed how to achieve reasonably 
accurate and higher-aggregated levels. 

[TNMOl] later addressed the multi-resolution question in federations, and identified 
the problems that arise from federations or interaction between simulations with different 
levels of detail or abstraction requiring aggregation and disaggregation of information re¬ 
quired by an HLA federation. [TNMOl] submit two new approaches: the middleware, and 
the federate-based approaches to replace the module-based approach to implement 
aggregation/disaggregation in HLA environments. 

Lacking integration, most model hierarchies are just a collection of models with dif¬ 
ferent resolution and some type of procedures for calibrating selected input parameters of 
the lower-resolution with higher resolution models. [Dav98] noted the different design phi¬ 
losophy between the normal bottoms-up approach favored by the Department, which as¬ 
sumes truth is inherent in the most detailed models, whereas MRM, as noted in Table 4-1 
supports any approach in achieving mutually calibrated models in a family. Today, with 
the end of the Cold War added increased uncertainty and added significant complexity to 
possible analytical issues and M&S development arising from the increasing instability, 
new variables, and unfamiliar post-Cold War geographical-political-military environments. 
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Figure 4-1. Resolution Dimensions (From [Dav98]) 

Different design methodologies listed in Figure 4-2, supported by several MRM 
tools exist. [DBMOO] described the implementation of a fast-running, stochastic, mul- 


96 




tiresolution, stochastic model PGM^^' Effectiveness Modifier (PEM) for exploratory analy¬ 
sis into the eomplex situational and tactieal factors of long-range preeision fires. In another 
effort to conduct insight-oriented exploratory analysis, [MDBOOb] described the effort to 
employ the PC-based EXHALE model, based on the analytieal methods identified in 
[DC97], to eonduct a study of interdietion eapabilities to quiekly stop an invading army. 
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Figure 4-2. Distinetion between Normal and MRM Design (From [Dav98]) 


In a related effort, the ARPA/Tri-Serviees sponsored the Rapid-prototyping of Ap¬ 
plication Specific Signal Processors (RASSP) program research by [HMB+00] reviewed 
and compared simulation terminology, interoperability challenges, and previous modeling 
taxonomies; as a basis for designing a multi-axis taxonomy to describe the information 
content of computer model types; and define abstraction levels supporting development of 
interoperable models. The RASSP modeling taxonomy provides a framework to categorize 
models based on a set of attributes, which may distinguish models for different intended 
uses, and establish formal, concise, unambiguous definitions for various model types which 

• Represent model attributes relevant to model designer and users, 

• Provide a readily understandable common terminology, 

• Identifies five distinct model characteristics: temporal detail, data value detail, func¬ 
tional detail, structural detail, and programming level. 
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• Consists of a set of attributes eharaeterizing a model’s relative resolution of details 
for important model aspeets shown in Figure 4-3 [HMB+00]. 
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Figure 4-3. RASSP Taxonomy Model (From [HMB+00]) 

Within the RASSP Taxonomy Model at Figure 4-3, a given model instanee de- 
seribes information, presented graphieally at one speeifie level within the model, even 
though some terms may span a range of abstraetion levels . The RASSP Taxonomy 
Model provides five eategories of resolution in whieh; 

• Temporal Resolution Axis represents the resolution of events presented on a time 
seale, 

• Data Resolution Axis represents the resolution of the format of values speeified in 
the model, 

• Functional Resolution Axis represents the level of detail of a model describing the 
functionality of a component or system. 
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The abstraction is an indication of the level of detail specified about how a function is to be implemented and is adversely related to 
the level of detail. If there is much detail, or high resolution, the abstraction is considered low, and the abstraction levels form a hierar¬ 
chy [HMB+00]. 
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• Structural Resolution Axis represents the level of detail of a model deseribing how 
a component is constructed from its eonstituent part, 

• Software Programming Resolution Axis represents the level of granularity of soft¬ 
ware instructions that the model of a hardware component interpret when exeeuting 
target software [HMB+00]. 

The level of abstraction does not indieate accuracy; in the same way preeision is 
different from accuracy, since two different models of a given function with different ab¬ 
straction may be equally as aecurate. In addition the RASSP Taxonomy Model provides 
two views of the attributes, the internal view as viewed from inside the model, and the ex¬ 
ternal view as viewed from the model’s interfaee boundary, eaeh view supporting the tem¬ 
poral, value, strueture, and function resolutions for a total of eight attributes enabling elar- 
ity and preeision, unlike many other taxonomies whieh may mix attributes [HMB+00]. 

The internal resolution references how a model deseribes the timing of events, functions, 
values, and structures of the elements contained within the models boundary, whereas the 
external resolution describes how a model defines the interface of the modeled device to 
other deviees [HMB+00]. 

The RASSP Taxonomy Model supports several hierarchies'^^: the functional or 
logical hierarchy, and the struetural of physieal hierarehy. The funetional hierarehy de¬ 
composes a system aecording to functions (e.g., receiver, register, transmitter), and the 
struetural hierarehy deeomposes a system aecording to physical structure (e.g., frames, 
racks, chassis) or logical relationships (e.g., data structures) [HMB+00]. 

Newer MRM theories include possible modifieations to the HLA RTI and possible 
use of components. [SKO+01] eompleted additional work on MRM methods and proposed 
the use of state variables of simulation objects or environments in an MRM simulation sys¬ 
tem to determine the resolution of the models. [ASOl] submitted a new HLA service, the 
multiresolution management service, using existing HLA services as a suggested addition 
to the RTI. [HBM96] proposes building multiple levels of resolution with defined ranges, 
into multiple independent models, as opposed to one supermodel, with the ability to re¬ 
move representations, whieh are not required to eomplete comparative analysis. In a simi- 
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Hierarchies in RASSP are a multi-level system supporting aggregation and decomposition, in which a node at a given level of the 
hierarchy may be represented by the set of its descendent nodes on the next lower level [HMB+00]. 
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lar concept, [CGG98] explained the JSIMS Military Modeling Framework initiative, a De¬ 
partment enterprise-wide Tiger Team initiative to develop Mission Space Objects formed 
with interoperable, multiresolution models from reusable and dynamically re-configurable 
eomponents to achieve a eommon interpretation within a synthetic battle space. 

H. INTEROPERABILITY AND HETEROGENITY IN THE C4ISR DOMAIN 

Interoperability poses significant challenges in many areas including the Command, 
Control, Computers, Communieations, Intelligence, Surveillance, and Reconnaissanee 
(C4ISR) dimension. [DoD02d] defines critical Department M&S requirements for im¬ 
proved C4ISR. However, there are currently severe M&S-to-C4ISR shortcomings identi¬ 
fied by [SFJ+98, RHS99, Sut99, Tol99, DAOO, TLW+00, MLOl, BBN+01, HMT+01, 
WHLOl, CWS02, DoD02e, DoD02d, ST02] for modeling information superiority. 
[DoD02d] summarizes these shortcomings and notes: 

• Good information superiority models do not exist, 

• Pre-seripted battlefield entities are non-responsive and lower study fidelity, 

• Existing models are simplistic, speculative, and too narrowly focused, lacking the 
ability to do end-to-end analysis, 

• Current force-on-force models do not aeeount for C4ISR or make broad assump¬ 
tions, 

• A lack of system performanee data, 

• The Department needs integrated treatment of threat signatures and sensor mission 
management, 

• Suitable Opposing Foree (OPFOR) representations are lacking [DoD02d]. 

A SISO M&S-to-C4I Interoperability Working Group report [TLW+00] identified 
that uncoordinated M&S and JTA standards have been, and eontinue to be developed by 
both eommunities, and several faetors inhibit a eompletely seamless M&S-to-C4ISR inter¬ 
face including: 

• Applicable standards (HLA, JTA) in both domains continue to evolve quickly, 

• Interoperability between C4ISR systems or eomponents may span multiple levels, 

• Eaeh domain must be able to retain authority and responsibility for data V&V and 
security, 

• Each domain has access point responsibilities for data management, metadata, and 
conversions, 

• Both domains must support active interfaces at the domain boundaries, 
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• Both architectures require a translation capability until future architectures employ 

the same data elements based on a common design [TLW+00]. 

The C4ISR Architecture Framework [AWG97] provides the strategic direction for 
C4ISR architecture development throughout the Department, and provides guidance on the 
development of a broad set of products used to document the operational, system and tech¬ 
nical architecture views. An M&S-to-C4ISR interoperability framework proposed by 
[TLW+00] includes a three-phase plan with near-, mid-, and long-term solutions evolving 
from a limited message-based as-is architecture. The to-be architecture shares common 
databases, implicit interoperability, “plug and play” capabilities, and full duplex informa¬ 
tion exchange [TLW+00]. 

[TLW+00] identifies success in the mid-term architecture with common compo¬ 
nents, improved interfaces, and common standards, with simulations updating C4I systems 
over two-way communications initialized by either the simulation or the C4I system. Cur¬ 
rently translators establish this interface, although translators represent an interim, expen¬ 
sive, error-prone, single-point-of-failure software solution until common, credible, interop¬ 
erable components become available. [TLW+00] proposes solutions including a C4I/M&S 
technical reference model (TRM), a broad data class (metadata), a reference FOM, and 
SISO guides to linking C4I to M&S. The Levels of Information Systems Interoperability 
(LISI) model [AWG98], shown in Figure 4-4 illustrates components or systems spanning 
multiple levels of sophistication, supporting various system-to-system information ex¬ 
changes. 

Current department data quality initiatives support the object-oriented approach. 
The U.S. Army developed the Army Object Model Standards Category (OMSC) [JHB98, 
HB98a. HB98b, MG98] for M&S domain objects, and the Army Integrated Core Data 
Model (AICDM) a standard reference data model [WHL+01], which includes the Joint 
Common Data Base (JCDB), the Army Land C2 Information Exchange Data Model, and 
the C2 Core Data Model [MHL+02]. [HB99, LFWOO, WHL+01, WHL+02] assessed in¬ 
teroperability between M&S and C4ISR data models, and evaluated the degree of align¬ 
ment to determine common/data object interoperability. 
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An Army M&S Interoperability Working Group [BunOO], reviewed the teehnieal 
ehallenges, eomplexity, eost, and eritieality of M&S interoperability, and provided a defini¬ 
tion for M&S interoperability^^"^ and proposed four types of interoperability within a ge- 
nerie 3-tier arehitecture for simulations including: 

• HLA Federation supporting federation runtime interoperability, 

• Scenario data sharing among different M&S (e.g., environment, weapons effective¬ 
ness data) including automated tools, and reduced cost, 

• Pedigreed data supporting the requirement for a lower-level M&S (e.g., higher reso¬ 
lution, precision) to develop data for a higher-level M&S in the Department’s M&S 
hierarchy, 

• Model / Algorithm sharing in M&S with different architectures and fidelity re¬ 
quirements [BunOO]. 

Related initiatives include the Department’s Data Architecture [JTA02a], interfac¬ 
ing with C4I systems [TolOl], the development of a DII COE M&S infrastructure [HSOOa], 
interoperability certification [BKW+02], object architectures supporting semantically in¬ 
teroperable systems [BCC+00], and weapons effectiveness [LPA+02]. [TH03] noted major 
discrepancies between the current OMSC and the JCDB standard reference data models 
with implications that the simulations currently under development do not support the 
Army Battle Command Systems, and require custom interface software or translators. 

A follow-on SISO M&S / C4I study group [BCC+02] confirmed the need for a 
common technical reference model (TRM) [TLWOO] to improve interoperability based on a 
review of M&S reference models, a prototype C4I / M&S interoperability TRM, informa¬ 
tion exchange activities, a general unified model (GUM), and related reference models. 
Proposed future initiatives included continued analysis of interoperability theories, a C4I / 
M&S TRM use-case study and development of TRM user guides. [TH03] proposed addi¬ 
tional challenges to the current C4ISR-to-M&S incompatibility issue with six theses he 
framed as grand-challenges. 

In addition, [TH03] propose to foster the development of information techniques 
and systems to support the war fighter based on the same views of the world, a common 
ontology, a common technical framework, and a common understanding of the constituent 
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dynamics. [TH03] also express concern with the pereeption that the M&S eommunity has 
not provided enough emphasis on the interoperability issues between C4ISR systems and 
M&S, beyond the development of representations of units, equipment, and behavior. 

As a result, [TH03] identified a need for a common architecture, a common ontol¬ 
ogy, a common set of algorithms and methods supporting a eommon overarehing concept 
bridging the different methodologies. The Department’s alleged M&S research void cited 
by [TH03] provides a major impetus for this dissertation and the development of product 
line domain arehiteeture. The six theses proposed by [TH03] to improve the C4ISR sys¬ 
tems and military simulation systems ability to support future military operations inelude; 

• Complementary teehniques needed for future military operations, 

• Use of the same eommercial standards, 

• Use of the same common architecture, 

• Use of data and objeet models based on the same common ontology, 

• Use of the same common set of algorithms and methods, 

• A common overarching concept for C4ISR systems, and military simulation sys¬ 
tems development and evolution [TH03]. 

Finally, [PaeOla] contends that although the Joint Teehnieal Arehiteeture (JTA) provides 
the Department arehiteeture for exehanging information at the current time, and the HLA 
defines interoperability standards for M&S, no standards currently exist for: 

• Decomposing M&S into entities and proeesses, 
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• Representation abstraction of the simulated subjeet, 

• Doeumenting authoritative representations in the M&S eoneeptual model [PaeOla]. 

I. SUMMARY 

Chapter IV reviewed several heterogeneity ehallenges to Department M&S eredibil- 
ity. Improving the credibility of large-scale, legaey Department M&S requires an under¬ 
standing of the underlying systemic causes generating the pre-eonditions for heterogeneous 
system representation anomalies, espeeially in federation interoperability. These diverse 
challenges inelude syntactic and semantic heterogeneity, data heterogeneity, complexity, 
interoperability, technical and substantive interoperability, fidelity heterogeneity, and mul¬ 
tiresolution modeling heterogeneity. 
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Authoritative representations have been eonsistently identified as major objeetives 
in Department M&S, yet few of the [DoD95, DoDOla] objeetives for authoritative repre¬ 
sentations have been achieved [MosOO, CraOla, CraOlb]. Senior civilian and military deci¬ 
sion-makers have significant reservations that Department M&S, as currently practiced, 
can meet the serious future demands generated by national security requirements for M&S 
in the Department. In addition, current Department M&S users still require accurate, inter¬ 
operable elements of the mission-space and the draft [DoDOla] retains authoritative repre¬ 
sentations as a Department-wide M&S objective including U.S., allied, friendly, coalition, 
neutral and threat and paramilitary system representations for C4ISR, combat, combat sup¬ 
port and combat service support systems and processes. Successful attainment of these in¬ 
teroperability objectives by the Department and international M&S communities assumes 
the acceptable resolution of many heterogeneity issues, including abstractions, networks, 
federations and representations. 

Abstraction is an intrinsic human activity employed to reduce or filter the full com¬ 
plexity of the real world to manageable proportions by deleting unnecessary detail in order 
to provide data in context (e.g., information) supporting the decision-making process 
[HJS97]. Abstraction mechanisms [Til98] selectively emphasize the level of detail, em¬ 
phasizing the details pertinent to the intended use, while hiding unneeded detail. [Sow88] 
discussed the three most common abstraction mechanisms classification, aggregation, and 
generalization [Sow88, Til98]. 

Data models establish the essential properties and well-defined relationships be¬ 
tween raw data in a system in a form, which supports efficient storage, timely retrieval of 
data, and provides the basis for tools and techniques to support data modeling. A data 
model captures the static and dynamic characteristics supporting data-related processes, 
with static properties defined in a database schema and the dynamic characteristics devel¬ 
oped into specifications for reports, queries, and transactions. The schema maintains the 
object type definitions, attributes, relationships and static constraints of the data repository 
or database, an instance of the schema. 

The most significant record-based logical data models include the hierarchical data 
model and the relational data model. These classic highly machine-oriented, and record- 

105 



oriented data models, or syntactic data models are well suited to the eomputer environment 
and organized for optimal efficiency supporting storage and retrieval operations, however 
these models lack semantic relativism and abstraction mechanisms required for dealing 
with complexity in large-scale systems [BCE+OO]. 

[Til98] describes a second category, semantic data models, which combine basic 
knowledge representation techniques with database technology, and represent a transition 
from the basic record-oriented approach towards a model supporting more human-oriented 
semantic constructs. While traditional data models store data, and semantic models shift 
towards a more human knowledge model, [Til98] suggests a third method, the conceptual 
modeling activity geared more towards the domain-level knowledge and program under¬ 
standing with a focus on the end-user. [Til98] further suggests that all three techniques 
may be more suitable than one method if there are different categories of artifacts (e.g., 
data, knowledge, information), requiring a relational model physical storage of data, a con¬ 
ceptual model for representing domain-level knowledge, or a semantic model for interac¬ 
tive discovery. 

[Wie93] acknowledged that data comes from many diverse and heterogeneous 
sources and further suggested that joining heterogeneous data is necessary to generate in¬ 
formation. Focusing primarily on database constructs and schema research, [WW90, 
Wie93, WieOOa, WieOOb, WieOl] developed eight viewpoints for differences among sys¬ 
tems, and [You02c] added a ninth viewpoint of heterogeneity germane to this research: 

• Heterogeneity of Hardware and Software systems, 

• Heterogeneity of Organizational Models 

• Heterogeneity in Representation 

• Heterogeneity of Scope, 

• Heterogeneity of Level of Abstraction, 

• Heterogeneity of Meaning, 

• Heterogeneity of Temporal Validity [Wie93], 

• Semantic Heterogeneity [WieOOa, WieOOb], 

• Heterogeneity of Structure [You02c]. 

This research revealed progress in the area of technical interoperability, including 
research into multilevel simulation of discrete network models [Ham96, BKG+02], distrib¬ 
uted simulations [Ham96, HNP97], and the Department-sponsored development of the 
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HLA. Although there has been progress with the teehnieal interoperability problem sinee 
the mid-1990s, many approaehes failed to improve the eredibility of authoritative represen¬ 
tations and reduce the federation’s interoperability problems caused by substantive interop¬ 
erability anomalies. Over time developers modeled many of the same objects in different 
ways, potentially causing problems when they interact in a federation. 

[DSB-l-99, HYOla, HYOlb, YPHOl] identified systemic substantive interoperability 
issues with legacy simulation software systems and the inconsistent representation of the 
same entity in multiple systems. There is a very close correlation on the root causes be¬ 
tween the technical and substantive interoperability issues cited in the simulation domain, 
and the previously cited work of [Wie93, You02c] on information from heterogeneous 
sources. However, substantive interoperability remains a systemic issue persisting through 
the evolution of the ALSP, DIS and HLA protocols, and remains a major challenge for 
achieving truly distributed simulation interoperability. 

[DB99] defines multiresolution modeling as “building a single model, a family of 
models, or both to describe the same phenomena at different levels of resolution” [DB99]. 
[DH92a] noted that achieving interoperability in diverse, independently developed M&S 
with different, but overlapping resolutions, based on different assumptions, limitations, and 
perspectives is a major challenge in contemporary M&S. [Dav92a, Dav93 and DBM99] in 
Figure 4-1 illustrate that the term resolution represents many different concepts and dimen¬ 
sions. However, [DavOO] noted MRM may not be sufficient for applications requiring dif¬ 
ferent abstractions or perspectives, that vary by conception of the system or use of vari¬ 
ables, and requires a new model construct for both multiple resolution and multiple per¬ 
spectives (MRMPM). 

In [MDBOOa] models and families of models employed hierarchical trees to remain 
mutually consistent across multiple resolution and perspectives. [BDMOO] employed an 
exploratory analysis technique using low-resolution models at the macro level of analysis 
and high-resolution models to support calibration and validation efforts. [TNMOl] later 
addressed the multi-resolution question in federations, and identified the problems that 
arise from federations or interaction between simulations with different levels of detail or 
abstraction requiring aggregation and disaggregation of information required by an HLA 
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federation. [TNMOl] submit two new approaehes: the middleware, and the federate-based 
approaehes to replaee the module-based approaeh to implement aggregation/disaggregation 
in HLA environments. In a related effort, the ARPA/Tri-Serviees sponsored the Rapid¬ 
prototyping of Application Specific Signal Processors (RASSP) program to review and 
compare simulation terminology, interoperability challenges, and previous modeling tax¬ 
onomies as a basis for designing a multi-axis taxonomy to describe the information content 
of computer model types and define abstraction levels supporting development of interop¬ 
erable models [HMB+OO]. 

Interoperability poses significant challenges in many areas including the Command, 
Control, Computers, Communications, Intelligence, Surveillance, and Reconnaissance 
(C4ISR) dimension. [DoD02d] defines critical Department M&S requirements for im¬ 
proved C4ISR. However, there are currently severe M&S to C4ISR shortcomings and: 

• Good information superiority models do not exist, 

• Pre-scripted battlefield entities are non-responsive and lower study fidelity, 

• Existing models are simplistic, speculative, and too narrowly focused, lacking the 
ability to do end-to-end analysis, 

• Current force-on-force models do not account for C4ISR or make broad assump¬ 
tions, 

• Models lack of system performance data, 

• The Department needs integrated treatment of threat signatures and sensor mission 
management, 

• Suitable Opposing Force (OPFOR) representations are lacking [DoD02d]. 

A SISO M&S-to-C4I Interoperability Working Group report [TFW+00] identified 
that uncoordinated M&S and JTA standards have been, and continue to be developed by 
both communities, and several factors inhibit a completely seamless M&S-to-C4ISR inter¬ 
face including: 

• Applicable standards (HFA, JTA) in both domains continue to evolve quickly, 

• Interoperability between C4ISR systems or components may span multiple levels, 

• Each domain must be able to retain authority and responsibility for data V&V and 
security, 

• Each domain has access point responsibilities for data management, metadata, and 
conversions, 

• Both domains must support active interfaces at the domain boundaries, 

• Both architectures require a translation capability until future architectures employ 
the same data elements based on a common design [TLW+00]. 
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An M&S-to-C4ISR interoperability framework proposed by [TLW+00] ineludes a three- 
phase plan with near-, mid-, and long-term solutions evolving from a limited message- 
based as-is arehiteeture. The projected to-be architecture shares common databases, im¬ 
plicit interoperability, “plug and play” capabilities and full duplex information exchange 
[TLW+00]. 
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V. 


THE DEPARTMENT’S FRAMEWORK FOR IMPROVED CREDIBILITY 


A. INTRODUCTION 

Chapter V provides the Department’s eurrent arehitectural framework for resolving 
the underlying systemie eauses generating the pre-eonditions for heterogeneous system rep¬ 
resentation anomalies, espeeially in federation interoperability. The Department’s initia¬ 
tives to improve the credibility of large-scale, legacy Department M&S supporting distrib¬ 
uted interoperability include the “as-is” architecture and the evolving “to-be” architecture: 
the Joint Technical Architecture, and the Common Technical Framework. The Common 
Technical Framework components include the High-Level Architecture, conceptual model 
requirements, and data standards supporting credible simulation development. 

B, THE DEPARTMENT’S AS-IS SIMULATION ARCHITECTURE 

The Department established terms of reference for an M&S framework to improve 
the communication of concepts, interoperability, and development of authoritative repre¬ 
sentations including architectural implementations (e.g., SIMNET, ALSP, DIS, HLA). The 
Common Technical Framework (CTF) for M&S contains three components: the HLA, 
CMMS, and data. The Federation Development and Execution Process (EEDEP) support 
the implementation of the HLA. The HLA is the current Department and IEEE standard 
software architecture for distributed M&S interoperability, although it has not fully re¬ 
solved long-standing technical and substantive interoperability issues. Consequently, rep¬ 
resentations of the same system in different models are frequently incompatible. 

Conceptual Models of the Mission Space (CMMS) / Eunctional Description of the 
Mission Space (FDMS) are the developer’s method for developing the M&S requirements 
into a detailed designed framework to develop the software. The verified and validated 
CMMS / EDMS subsequently supports the critical role of verifying the software develop¬ 
ment effort. Authoritative data, the final component of the CTF is critical to the Depart¬ 
ment’s goal of improving credibility and confidence in simulation results, however authori¬ 
tative data may be the single most pervasive M&S deficiency area. 
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Software architecture theory continues to evolve in academia, the private sector, 
and the public sector including the Department. Prior to the development of the HLA, the 
Department’s M&S as-is distributed architecture framework consisted of the Simulation 
Network (SIMNET), the Aggregate Level Simulation Protocol (ALSP)^^^, and Distributed 
Interactive Simulation (DIS) environments. SIMNET [MCU+95, Gre99] was a pioneering 
collaborative effort from 1983 until 1989 between the Defense Advanced Research Projects 
Agency (DARPA) and U.S. Army, to develop distributed simulations operating on several 
interconnected computers and determine the feasibility for distributed simulations to pro¬ 
vide a training capability for the U.S. Army. 

The ALSP distributed network protocols patterned after the SIMNET effort 
[WSW9I, WWG93] permits multiple, pre-existing, large-scale aggregated-level, construc¬ 
tive warfare simulations to interact in logical time over local or wide area networks, control 
its own objects and share information with other simulations. ALSP, originally developed 
by DARPA in 1992, and currently managed by the U.S. Army Simulation, Training and 
Instrumentation Command (STRICOM) through a multi-service agreement, was based on 
four principles, and a distributed architecture composed of a three-part, two-layer proto- 
col component and two software components, permitting interoperation of dissimilar 
Service and Joint constructive M&S [WSW91, WWG93]. 

Early ALSP confederations of Service simulations (e.g.. Corps Battle Simulation, 
Air Warfare Simulation) supported several joint and combined training exercises (e.g., At¬ 
lantic Resolve, Unified Endeavor, Ulchi Locus Lens) [DoD95]. The Joint Training Con¬ 
federation, initially deployed for major training exercises in 1992, evolved into the largest 
ALSP confederation with eight interacting simulations [Log02]. The ALSP confedera¬ 
tion simulations, developed originally as stand-alone systems, support only limited interop¬ 
erability, require lengthy set-up times, lack interfaces to real-world systems, and need large 

ALSP is a family of M&S interface protocols and supporting infrastructure software that permits the integration of distinct M&S and 
war games. Combined, the interaction protocols and software enable large-scale, distributed M&S and war games of different domains 
to interact at the combat object and event level [DoD98]. 
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ALSP principles: 1) Distributed computation based on combat entity ownership, 2) avoidance of single critical resources, 3) reliance 
on broadcast communications, and 4) replication of a limited set of combat entity attributes among all M&S [WSW91]. 

The ALSP protocol component exists of 1) a peer level protocol joins translators and object attribute management, 2) a connection 
protocol manages the translator to gateway, and 3) a second peer level protocol that joins gateways and deals with timing issues. The 
ALSP software components include the translators and the gateways [WSW91]. 

™ ALSP interacting simulations: AWSIM, CBS, RESA, MTWS, CSSTSS, JQUAD, TACSIM, and PSM [Log02]. 
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amounts of manpower. The Department plans to replaee ALSP eonfederations with the 
Joint Simulation System. 

The Distributed Interaetive Simulation (DIS) protoeol [MCU+95, IEE98a] provides 
a synthetie interaetive networked environment in whieh humans may interact real time, 
through the connection of different platform-level virtual M&S, simulators, or instru¬ 
mented live exercises. The DIS protocols and standards [IEE98a] establish a common data 
exchange environment or common messaging environment, using Protocol Data Elnits, 
supporting the interoperability of heterogeneous, geographically-distributed live, virtual, 
and constructive simulations [DoD95]. The DIS synthetic environment uses verified sce¬ 
narios, tactics, techniques, and procedures to train testers on new hardware or software 
[AWQ+93], and conduct trial test runs before executing costly field tests. 

The DIS protocol development effort also experienced broad participation from an 
open forum of industry, government, and academia, and plays a central role in the Depart¬ 
ment’s M&S portfolio. [EBV96, Gre99] cited key DIS techniques including a “dead reck¬ 
oning” methodology based on a world and body coordinate system to minimize communi¬ 
cation requirements between simulators in order to produce a realistic and credible virtual 
battlefield. However, the computation-intensive DIS protocol requires high bandwidth lev¬ 
els for it’s broadcast messaging technique, and lacks support for different time management 
methods, realistic command and control representations, and dynamic changes in the envi¬ 
ronment [DoD95]. 

C. THE DEPARTMENT’S TO-BE ARCHITECTURE 

The Department’s future “to-be” architecture development includes the evolving 
Joint Technical Architecture (JTA), common Operating Environment (COE) and C4ISR 
Eramework supported by the Technical Reference Model (TRM) with a major focus to im¬ 
prove interoperability and information exchange. The Department’s COE, JTA, and 
C4ISR framework support the foundation of the software-intensive “to-be” architecture. 

The Department’s “to-be” architecture or target architecture will evolve to meet the per¬ 
formance requirements codified by the Government Performance and Results Act of 1993 
and Clinger-Cohen Act of 1996 [CC96], the legal frameworks for developing United States 
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Government enterprise-level information teehnology arehiteeture standards and poliey 
[LSF+Ol]. The Department’s “to-be”-architeeture will also reflect the emerging Federal 
Enterprise Architecture standards identified by [LLT-l-99] and approved by the Chief In¬ 
formation Officer (CIO) council [HamOOa] under the direction of Executive Order 13011, 
for Eederal Information Technology. 

The Department’s Enterprise Integration (El) Implementing Strategy and Enterprise 
Model described by [DoDOOc] identifies near- and long-term strategies including the exist¬ 
ing Corporate Information Management strategy, an integrated technical architecture 
framework for information management, data standards, and shared databases. The long¬ 
term El strategy provided by [DoDOOc] “ensures consistency, quality, timeliness, availabil¬ 
ity, and security of shared, corporate data by implementing corporate databases using stan¬ 
dard data elements as soon as possible” [DoDOla]. 

The Defense Information Infrastructure (DII) Common Operating Environment 
(COE) [Har97, Har98a, DoDOOd] emerged in the mid-1990s as a foundation for developing 
open systems with a “plug and play” open-architecture capability based on the open techni¬ 
cal architecture developed for the Global Command and Control System [HPS98]. The DII 
COE, an architectural approach for building interoperable systems, includes a collection of 
reusable components, a software infrastructure, and a set of standards and guidelines for an 
open architecture designed around a client / server model [BPM-l-95]. The DII COE also 
contains products to meet operational requirements, which are not fully JTA compliant 
[HBOO, JTAOla]; however, the goal is to evolve to full compliance with the appropriate 
JTA standard. In support of this goal, [HSOOa, CHOI, CMOS] presented M&S infrastruc¬ 
ture alternatives to better align the current M&S architecture with the DII COE. 

The Joint Technical Architecture (JTA) [JTA02a, JTA02b] and the C4ISR Architec¬ 
ture Framework [AWG97] provides the Department with the basis for developing systems 
with the required seamless interoperability, defining service areas, interfaces, terminology, 
and standards applicable to all systems and mandated for the management and development 
of new or improved Department systems [Ste02a]. The Department implemented the JTA 
by defining an interrelated set of operational, technical, and system views. The JTA also 
builds on service areas identified in the Technical Reference Model [BKOla, TRMOla, 
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TRMOlb], supported by technically stable, mature commercial standards and guidelines. 
The JTA [JTA02a] consists of two main parts: the JTA Core and the JTA Domains with 
sub-domains shown in Figure 5-1, the JTA Hierarchy Model. 



Figure 5-1. JTA Hierarchy Model (From [JTA02a]) 


The JTA [JTA02a] core information technology categories include information 
processing standards, information transfer standards, information modeling, metadata, in¬ 
formation exchange standards, human-computer interface, and information security stan¬ 
dards. The JTA also identifies the minimum set of JTA elements applicable to all Depart¬ 
ment systems to achieve interoperability and domain-specific elements to ensure interop¬ 
erability within the domain (e.g., C4ISR, Combat Support, Modeling and Simulation, and 
Weapon Systems), but not necessarily for inter-domain interoperability between the four 
JTA domains or their subordinate domain elements [JTA02a]. 

The Department’s Technical Reference Model (TRM) [TRMOla, GMWOOb] and 
TRM User Guide [TRMOlb] support the requirements of increasingly complex and diverse 


115 




systems by integrating the service and interface views, and identifying interfaces and con¬ 
tent. The [TRMOla] evolved from the Department’s Technical Architecture Framework 
for Information Management (TAFIM), and includes three major model elements (1) the 
services, (2) the interfaces and (3) the entities. The three major model elements specified 
by the [TRMOla] include the following characteristics: the ability to support system archi¬ 
tectures [Dem95b, MR02], model degree of freedom to select or expand on services and 
interfaces, and the ability to support new services. 

The [TRMOla] also provides interface definitions, and environment configurations, 
supports a model’s ability for different views, and provides methods of mapping the model 
to other known reference models. In a related effort within the M&S domain, the SISO 
C4ISR/ Simulation Technical Reference Model study group [BBM+01, GLT+02] devel¬ 
oped a standard frame of reference for interoperability between C4ISR systems and M&S 
systems. [RHS99] also developed a supporting conceptual model with an information ex¬ 
change model focused on the information exchange between C4ISR and M&S components. 

The Department developed the JTA [JTA02a, JTA02b] and the C4ISR Architecture 
Framework [AWG97] to improve the system architecture. However, [CBB+03] caution 
that the C4ISR Architecture Framework [AWG97] almost exclusively documents the sys¬ 
tem architecture, and that none of the three [AWG97] views (e.g., operational, systems, 
technical), or the essential or supporting products, “prescribe anything that remotely re¬ 
sembles software architecture” [CBB+03]. This assessment of the maturity of the Depart¬ 
ment’s software architecture including [AWG97] is a systemic issue, since [lEEOOb] notes 
there is no single, accepted framework codifying software architectures, despite significant 
research in the area. 

The Modeling and Simulation Domain prescribed by [TRMOla] includes the De¬ 
partment’s Common Technical Eramework (CTF) [DoD95, GMS+96] for facilitating M&S 
interoperability and reuse. The CTF has three components: the High-Eevel Architecture 
(HEA) to which the Department’s which M&S must conform; Conceptual Model of the 
Mission Space (CMMS) to provide a basis for the development of consistent and authorita¬ 
tive representations; and data standards to provide common representations of data across 
M&S and C4I systems [DoD95, GMS+96]. The Department’s M&S architecture efforts 
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cited by [DoD95, GMS+96] focused on interoperability and improving the shortfalls of the 
previous interoperability protocols (e.g., ALSP, DIS). 

1, High-Level Architecture (HLA) 

The [DoD95] defined the requirement for a common high-level simulation architec¬ 
ture to facilitate interoperability of all types of simulations, interoperability with C4I sys¬ 
tems, and improved reuse of M&S components. In the Department’s M&S domain, the 
current standard software architecture for distributed simulation interoperability is the 
HLA. Under development since 1995 and designated the technical architecture for all De¬ 
partment M&S [GanOO], the HLA technical architecture includes three major components: 
the simulation member of the federations, or federate, the runtime infrastructure (RTI), and 
the runtime interface. The HLA, is also an IEEE standard [lEEOOc] with a reusable com¬ 
mon distributed framework, composed of an HEA rule set [IEE98e, lEEOOc], the HLA in¬ 
terface specification [lEEOOd,] and the HLA Object Model Template (OMT) [lEEOOe, 
LutOO], supporting a wide range of M&S application areas at different resolutions. 

The HLA rules [lEEOOc] are divided into two major categories: federate and federa¬ 
tion rules. An HLA federate may be another computer-based simulation, a manned simula¬ 
tor, an interface to a live or instrumented range, or a simulation utility. An HLA federation 
is a set of interacting simulations or federates represented by a federation object model 
(EOM) based on the OMT format [lEEOOc, BRA02]. The HEA OMT [lEEOOe] specifies 
the object model and the essential objects, attributes, and interactions, shared across the 
federation, supporting interoperability. An HEA federate requires a simulation object 
model (SOM) based on the OMT, to identify all public information shared across the fed¬ 
eration [lEEOOc]. 

A EOM identifies the essential classes of objects, object attributes, and object inter¬ 
actions [lEEOOe], supported by the HEA federation. At runtime in an HLA federation, 1) 
the federate manages all object representations, and 2) only one federate owns any speci¬ 
fied attribute of an instance of an object at one time. The HLA interface specification de¬ 
scribes the run time services provided by the RTI to a federate, and prescribes the interface 
from the federate to the RTI, with six classes of management services: federation, declara- 
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tion, object, ownership, time (continuous, time-stepped, discrete event), and data distribu¬ 
tion [lEEOOd]. 

In addition, the HEA supports the passive collection of data and the interface to live 
participants using the HEA interface to interact with the RTI [Dah99]. The HEA runtime 
interface specification supports the following RTI activities: I) a standard for federation 
interaction with the RTI, 2) the method to invoke RTI services supporting federation run¬ 
time interactions, and 3) the ability of a federate to interact with the RTI [lEEOOd]. Several 
HEA interoperability capabilities and lessons-learned identified by [Tol02] have been ad¬ 
dressed in federation calibration experiments [BLR-l-02], federating time-stepped and dis¬ 
crete event simulations [Bee02], and object-oriented architectures [CA02]. [Dah99, 
HarOlc] described the current technical status of the HEA and identified future challenges. 
Other initiatives include component-based extensions [RPK02]; employing the Extensible 
Markup Eanguage (XME) [SBOO] and Unified Modeling Eanguage (UME) methods sug¬ 
gested by [BRJ99, OMG99, SBOl, Oes02, and MB02] to improve HEA documentation; 
and distributed test and evaluation implementations [Har98b]. 

The HEA RTI according to [Dah99, CA02] acts as the distributed operating system 
providing the interface specification and management services to the federation. [BJ98] 
compared the HEA with other private sector distributing computing efforts including the 
Common Object Request Broker (CORBA) sponsored by the Object Management Group 
(OMG), and the Remote Method Invocation (RMI) from SunSoft’s Java Development Kit 
(JDK) [WWN97], and identified the strengths and weaknesses of each approach. [BCB02] 
discussed methods to manage, monitor, and manage federations. Viewing federation per¬ 
formance as a critical factor, [EMF02] explain RTI benchmark studies comparing three RTI 
implementation approaches, while [HW02] described the possible implementation of data 
distribution management services in the next-generation RTI. 

Community researchers continually provide recommendations for HEA improve¬ 
ments and enhancements. [HHS+98] suggests specifying additional optional classes of in¬ 
formation allowing the development of a more complete description of the federation struc¬ 
ture or behavior. [HHS+98] also explains the capstone definition, class terminology, and 
classification rules of a reference EOM. [RDOO] describe several additional EOMs, devel- 
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oped according to the format described in the HLA OMT, including a prototype C4I FOM. 
[GRH+01, GHL+01, and GHO+01] describe and specify the Base Object Model (BOM); a 
SISO-approved type of reference FOM designed to provide a component overlay capability 
to the HLA architecture. Other FOM initiatives include an intelligence and electronic war¬ 
fare FOM developed by [WS02] to bridge constructive, virtual and live systems in a syn¬ 
thetic environment to improve interoperability; a federation object model for atmospheric 
dispersion proposed by [HCP+02]; and a real-time platform reference FOM suggested by 
[OF02]. 

In order for the Department to achieve M&S interoperability goals, [MCL+99] 
identified several issues including the need for “a better means of agreeing and translating 
alternative model data representations (e.g., model interoperability)” [MCL+99]. In sup¬ 
port of improved simulation interoperability, [LLP+02] conducted and analyzed calibration 
exercises to baseline performance data, while [CCB+02] described the development of an 
HLA-compliant high-performance computing RTI to improve RTI interoperability per¬ 
formance issues identified by [MCL+99, FMF02]. The FOM and OMT (and their relation¬ 
ship with the RTI) also need extensions and better definitions according to [MCL+99]. 

After initial HLA federation experimentation and prototype efforts experienced a 
great deal of trial and error, the Department M&S community determined the need for ad¬ 
ditional guidance to improve cross-domain interoperability and future federation collabora¬ 
tion. The community achieved consensus for various “best practices” aspects of federation 
development, leading to the establishment of a common process view for HLA federations 
addressing the spectrum of interests within the HLA community, and introduced the Fed¬ 
eration Development and Execution Process (FEDEP) [DMS99] in September 1996, fol¬ 
lowed by a release of the EEDEP concept of operations in 1997. 

The EEDEP six-step process illustrated in Eigure 5-2 continues to evolve as the 
community experience identifies improvements [Wai97, Wai02a, DMS99] and efforts are 
currently underway to propose EEDEP IEEE guidance for future HLA federation construc¬ 
tion methods [LDG+02]. In a continuing trend to perform only inherently governmental 
functions, the Department transitioned the RTI development responsibilities from a 
DMSO-sponsored, government-funded project to the private sector for future development 

119 



and capitalization in October 2002. In December 2003, researehers at the Johns Hopkins 
Applied Physies Laboratory (APL) reported the first sueeessful eommereial applieation of 
the IEEE 1616 HLA speeifieation in a medical federation between the Massaehusetts Insti¬ 
tute of Teehnology eardiovascular system simulation and the APE’s detailed model of both 
the left and right ventrieles [JHU03]. 


PROGRAM AVAILABLE 

OBJECTIVES RESOURCES 



Eigure 5-2. The EEDEP Proeess (Prom [DMS99]) 

2. Conceptual Model of the Mission Space (CMMS) 

A conceptual model [DoD95, GMS+96] is the developer’s method of translating the 
M&S requirements into a detailed design framework to develop the software, and describes 
the conceptual model components, interactions, and the M&S concept of operations. Sev¬ 
eral researchers, including [SPC+98, PacOOa, SHOO, SheOOb, DDHOl] validated the basic 
need for conceptual models, and developed supporting perspectives of conceptual modeling 
theory, identified systemic issues, and proposed solutions. However, the Department’s 
management of conceptual models has two major challenges to overcome, an inconsistent 
past and a demanding future. 

Today, the Department’s M&S community currently lacks consensus and standards 
for conceptual models. Moreover, conceptual models are critical to the key Department’s 
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M&S issues of authoritative representations, improved simulation credibility, enhanced 
confidence in simulation results, integrated simulation security, and improved simulation 
interoperability. Surveying the current M&S situation, [LRH+01] consider the term “con¬ 
ceptual model” extremely overloaded and overused within the M&S community. For ex¬ 
ample, the term “conceptual model” describes the first abstraction of representations in an 
M&S and describes a high-level design of how all the components in an M&S relate to an¬ 
other. The wide and varied body of literature on the subject of conceptual models is an in¬ 
dication of their importance, but also reveals a lack of consensus on conceptual model 
methods, verification and validation, format and development methodology. 

A short survey on the breadth of conceptual modeling research, concepts, theories, 
and the different perspectives include the following widely-diverse conceptual model top¬ 
ics: verification and validation [Tho97b, Wai97, Pac98a, Pac98c, SPC+98, Pac99a, 
PacOOa, PacOOb, PacOOc, MetOO, LRH+01, PacOla, Bor02], conceptual model object- 
oriented schemas [Wai97], conceptual model quality [TB97], and conceptual model docu¬ 
mentation [Tho97b, Pac99a]. The literature also includes significant information on: con¬ 
ceptual model environment [Bir98], conceptual model aggregation / disaggregation 
[BidOO], fidelity [Pac98c, PacOOa, PacOOc, PacOla], conceptual models and the mission 
space [MM98], conceptual model validation [Pac98c], conceptual model/ functional de¬ 
scription of the mission space transition [HM98, JorOl], multiple conceptual model ap¬ 
proaches [Whi99], and conceptual model reuse [PacOOc, Pac02a]. In addition the literature 
search revealed addition citations for aircraft conceptual models [ChaOO], assumptions 
document [LKOO], joint conceptual models [Bir98, MetOO, LRH+01], and conceptual mod¬ 
els for HLA federations [Tho97b, LC97, Wai97, Pac98c, Bir98, PacOla]. 

The conceptual model may also include the algorithms, equations as well as any as¬ 
sumptions and limitations. However, even the naming convention for the Department’s 
conceptual models is challenging with the CMMS [DoD95, GMS+96] term and the newer 
Functional Description of the Mission Space (FDMS) [RPGOO] term used interchangeably. 
In this effort we will normally use the use the term conceptual model unless specifically 
addressing the CMMS- or FDMS-versions of the conceptual model methodology. 
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However, whiehever term or format is used, whether its is the older CMMS term or 
the newer FDMS label, [GMS+96] eites a Department’s key coneem with M&S eredibility 
as the laek of a detailed and doeumented conceptual model and design specification for ex¬ 
isting legacy systems M&S. Conceptual model verification is the Department’s process for 
ensuring the conceptual model meets specified requirements. However, the conceptual 
model documentation for many legacy M&S may be inadequate, if it exits at all, according 
to [Pac99b]. The conceptual model for simulations suggested by [PacOOa] addresses the 
simulation’s requirements, context, entities and processes. 

[Had98] proposed a specification of metrics for describing existing CMMS and for 
guiding subject matter experts in the acquisition and documentation of the Military Opera¬ 
tions Mission Space. [FY97] decomposes his conceptual model design and M&S imple¬ 
mentation according into a layered structure to account for model dependency, information 
flow, and model control; and envisions three different abstractions: the real world, the con¬ 
ceptual model of the mission space and the software implementation. Additional perspec¬ 
tives on the definition and purpose of conceptual models, where conceptual models fit into 
the M&S development lifecycle, and how conceptual models are developed include con¬ 
cepts forwarded by [GMS+96, Hom+97, SPC+98, DMS98a, DMS98b, DMSOOa, PacOOa, 
RPGOO, Bor02, Pac02d]. A properly developed conceptual model for the M&S is the best 
generic validity referent according to [PacOlb], since it captures both pertinent elements of 
system theory and intended M&S application. 

Today, there still exist several systemic impediments to the development of credible 
explicit conceptual models. Two major issues addressed by [Pac02a] are the lack of formal 
requirements or standards for conceptual models and the lack of an explicit M&S concep¬ 
tual model as a defined contract deliverable. There are also issues identified by [PacOOa] 
with abstracting representation from available information, or for describing and document¬ 
ing the conceptual model. In addition, [Bor02] is concerned about the level of understand¬ 
ing and development of conceptual models of the mission. 

Without a verified and validated conceptual model (e.g., CMMS or FDMS) upon 
which to base the verification of the software implementation, and ultimate validation of 
the simulation, it may be extremely difficult, if not technically impossible to credibly fol- 
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low the Department’s recommended guidelines for verification and validation [GMS+96, 
RPGOO]. This may result in an adverse opinion of the simulation’s credibility, which un¬ 
dermines user confidence in the simulations results. 

Definitions of conceptual model validity, model verification, operational validity, 
data validity, and recommended procedures forwarded by [Sar98] include the relationship 
of V&V to the development process. Models based on repeatable, measurable phenomena 
are usually empirical according to [Mey98] while simulations are normally probabilistic in 
nature since the supporting phenomena may be unknown, uncertain or questionable. 
[Mey98] also proposed the possible valid uses of a simulation if the real-world phenomena 
are unavailable, while acknowledging the possible invalid use of a model without knowing 
the supporting phenomena. [PacOla] also provides ideas on how to perform validation as¬ 
sessments when there is inadequate information about one or more federate conceptual 
models. 

[Had98] describes the problem of defining an appropriate level of resolution for the 
CMMS, and addresses ways that system purpose and desired capabilities can drive the de¬ 
velopment of prescribed requirements for CMMS. In a follow-on SISO-sponsored effort, 
[Gro99 and RGHOO] proposed a new CMMS methodology to define the context for the de¬ 
velopment and support of simulation object models and federation object models, based on 
describing and developing several CMMS corresponding to the battlespace of real mission 
areas, including sufficient information to support the new CMMS role as a referent for fi¬ 
delity measures. The overarching CMMS framework envisioned by [Gro99 and RGHOO] 
would provide access to the necessary detailed data and support the development of consis¬ 
tent and interoperable authoritative representations within a standard fidelity framework. 
More importantly an expanded CMMS model provides a possible solution to the systemic 
issue of how to define a standard fidelity referent, which has been a major obstacle to the 
development of accepted definitions of fidelity, which [Gro99 and RGHOO] contend is 
needed for supporting fidelity-based verification and validation process. 

Conceptual models also support the knowledge acquisition process and provide a 
common starting point for constructing consistent and authoritative M&S representations. 
The knowledge acquisition alternatives addressed by [TS92, CPS+96, Ede97, Mai97, 
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DRC99, KB099, DSB+01] support the development of a valid conceptual model. A con¬ 
ceptual model framework endorsed by [PacOOa] identifies four steps in developing a con¬ 
ceptual model, and proposes greater use of formal methods to develop M&S requirements 
and define the conceptual model. [HOM+97, RPGOO] specify a CMMS technical frame¬ 
work providing: 

• Technical standards, 

• Administrative procedures, 

• Operational infrastructure 

1 OA 

• The common semantics and syntax for describing the mission space, 

• A common format database management system [RPGOO], 

• A closed-loop engineering process for creating and maintaining conceptual models, 

• Data interchange format (DIF) standards for conceptual model integration and 
interoperability [HOM+97, RPGOO]. 

Several DMSO sponsored reports [HSB98, SheOOb, HSOOa, HSOOb] address the 
state of current conceptual model development methods and tools developed by and for the 
Department’s M&S community. A recent comprehensive assessment of the DMSO 
CMMS support effort by [LuqOl] addressed the following areas: a) the effectiveness of 
software risk assessment models, b) enhancements to the FDMS resource center, c) XML 
and wrapper-based translators for Department databases, and d) metrics for selecting auto¬ 
mated test tools; and suggested improvements based on the latest research. 

Most experts agree on the need for conceptual models, although a wide variety of 
interpretations and a great deal of confusion for conceptual modeling exists within the De¬ 
partment’s M&S community today; with consensus difficult to achieve on the exact defini¬ 
tion of conceptual modes [LRH+01]. As a result, the current draft [DoDOla] cites a need 
for developing new and improved V&V technologies and methodologies, and improve¬ 
ments for the V&V of representations. 


Every language of communication possesses two identifiable properties, the form of the language and the meaning associated with 
the form. The syntax of the language is a set of rules specifying which forms of the language are grammatically acceptable, its grammar. 
The meaning derived from a syntactically correct instance of a language may be viewed from two points, the meaning intended by the 
originator, the semantics of the language, or the meaning interpreted by a receiver, the pragmatic meaning [RR83]. 
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3. 


Functional Description of the Mission Space (FDMS) 


The term Funetional Deseription of the Mission Space (FDMS) [RPGOO, DouOl] 
replaced the previous Conceptual Model of the Mission Space (CMMS) term, in the De¬ 
partment’s M&S lexicon, as a required component of the M&S development and V&V 
processes. The FDMS provides the real-world description of entities, processes, and activi¬ 
ties for the design phase, and detailed representations of the problem domain in the re¬ 
quirements development phase of the simulation life cycle. Most of the conceptual models 
[LRH+Ol] described support classification either as a domain-oriented classification, such 
as the battlespace [PRM97], or design-oriented classification. There are four basic formats 
to describing conceptual models identified by [RPGOO]; the ad hoc method, design ac¬ 
commodation, development paradigm, and the scientific paper approach. The FDMS em¬ 
ploys the development paradigm method, although the [RPGOO] recommends the scientific 
paper. The ad hoc approach is the most common format employed today to develop the 
conceptual model [RPGOO]. The M&S developer normally produces the FDMS. 

4. Data Standards 

Data standards are the third component of the common technical framework for 
M&S. In any M&S application, the associated data quality should be as creditable as the 
M&S itself if the user or sponsor is to attach confidence to the results. [DoD95] defines 
data quality for the Department’s M&S programs. A proposed draft [DoDOla] places 
increased emphasis on data quality adding, “That a model implementation and its associ¬ 
ated data accurately represents the developer’s conceptual description and specification” 
DoDOla]. [BCE-l-99] also suggests data quality attributes including source accuracy, fidel¬ 
ity, suitability, credibility, maturity, cost, availability, and similar characteristics. [Kil02] 
recommends data accuracy and quality information supported by a review at three levels: 
database, data element, and data value; employing descriptive, quality and usage metadata. 


Data quality includes the correctness, timeliness, accuracy, completeness, relevance, and accessibility that makes data appropriate for 
use. Quality statements are required for source, accuracy (e.g., positional and attribute), up-to-dateness / currency, logical consistency., 
completeness (e.g., feature and attribute), clipping indicator, security classification, and releasibility [DoD95]. 
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A data verification and validation process assesses the quality of the data, employing data 
quality measures, and establishes the basis of confidence in the data [Kil02]. 

For each phase or aspect of the simulation’s design, development and operation, 
possible sources of performance information include live tests, data centers, ground tests or 
analytical M&S. Theoretically, the data producer uses metadata to describe the quality of 
the data as a part of data V&V process to meet the specification requirements and provides 
data meeting the user’s specification. The user assesses the applicability and quality of the 
data for a specific application based on the producer’s verified metadata. As part of the 
users V&V process, the data verification process checks the model algorithms, and as the 
last step, validates the data and algorithms to establish overall M&S credibility. The 
DMSO-sponsored Authoritative Data Source (ADS) project [VED98] identified seven 
hundred and fifty-six authoritative data sources that met ADS data procedures, including 
metadata definitions, in an effort to decrease the redundancy of data development, and im¬ 
prove data quality, fidelity, and data sharing. 

Environment data is critical to the M&S community. Sun Tzu realized the impor¬ 
tance of knowing the environment when he wrote, “Know the other, know yourself and the 
victory will not be at risk; know the ground, know the natural conditions, and victory can 
be total” [TzuVl]. The second [DoD95] objective included the development of natural en¬ 
vironment representations sub-objectives for terrain domains including: (2.1), ocean (2.2), 
atmosphere (2.3), and space (2.4) instantiated by the Ocean, Atmosphere and Space Envi¬ 
ronmental Services (OASES) System [RIOOl]. 

However, quality environmental data is hard to acquire. [BPT96] studied the eight 
most-needed types of environmental data identified by earlier surveys and concluded that 
the capability to acquire these data, especially at high fidelity were insufficient to meet re¬ 
quirements, especially in the models of fog, aerosols, humidity, visibility, and databases. 
Responding to these concerns, the Department invested heavily in environmental data. In 
an early DMSO sponsored initiative to improve environmental representations, the Naval 
Research Laboratory [NRL95] developed the Environmental Effects for Distributed Inter- 
active Simulation (E DIS) program. The E DIS program employed an object-oriented 
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methodology described by [HJG94, Hec94, WH94, AFW+94] to support distributed inter¬ 
active M&S with environments and environmental effects. 

The Synthetic Environment Data Representation and Interchange Specification 
(SEDRIS) program is a major Department follow-on initiative to the E DIS program with 
objectives cited by [SED98a, SED98b, SED98c] to improve environment representations. 
The SEDRIS vision [SED98a] is to provide a common synthetic environment to reduce 
cost, improve reuse, support interoperability, and provide effective data transfer between 
heterogeneous systems, initially in the training functional area. SEDRIS employed a meta¬ 
model based transmittal methodology with a standardized data model and supporting appli¬ 
cation program interfaces. The SEDRIS Data Model, based on an Object Model Template 
(OMT), with minor extensions, employs the concept of a class of data [SED98a] supporting 
a common environmental architecture. 

SEDRIS’ major function is to serve as an interchange medium supporting diverse 
requirements for different customer domains [Bir97]. SEDRIS also includes a Geospatial 
Reference Model (ISO 18024) supporting the specification of coordinates, datum, projec¬ 
tions, and several geo- and non-geo-referenced spatial reference systems, the SEDRIS 
transmittal format (ISO 18026) and the Environmental Data Coding Specification (ISO 
180230) [RMJ+02, JMP02]. Complementing these initiatives are current or projected ef¬ 
forts to acquire science data from other the private sector, international scientific programs, 
or cooperative public-private partnerships. 

NASA led the effort to collect data from space-based sensors, and continues to this 
day developing spacecraft for a complete Earth Observing System, with planned experi¬ 
mental Pathfinder missions under the auspices of the Earth Science Enterprise [PEE+00]. 
Another emerging international initiative collaborative effort promoting efficient geospatial 
data development, sharing and use is the Global Spatial Data Infrastructure [LWK+02]. 

However, remotely sensed data from space [JBS+93, NAB+92, EG99] may be dif¬ 
ficult to share based on different sensor characteristics (e.g., sensitivity, spectral coverage, 
calibration), the lack of standards, proprietary data processing systems and algorithms, and 
subtle variations that may adversely influence data consistency [JSK02]. In addition, en¬ 
suring commercial data sources are available requires an understanding of negotiations for 
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intellectual property [IEE99b, ATEOl] and understanding of how information technology 

1 ^7 

standards emerge, survive and evolve [EibOl]. 


D, SUMMARY 

Chapter V provided the Department’s current architectural framework for resolving 
the underlying systemic causes generating the pre-conditions for heterogeneous system rep¬ 
resentation anomalies, especially in federation interoperability. The Department’s initia¬ 
tives to improve the credibility of large-scale, legacy Department M&S supporting distrib¬ 
uted interoperability include the “as-is” architecture and the evolving “to-be” architecture: 
the Joint Technical Architecture, and the Common Technical Eramework. The Common 
Technical Eramework components include the High-Eevel Architecture, conceptual model 
requirements, and data standards supporting credible simulation development. 

The Department established terms of reference for an M&S framework to improve 
the communication of concepts, interoperability, and development of authoritative repre¬ 
sentations including architectural implementations (e.g., SIMNET, ALSP, DIS, and HEA). 
The Eederation Development and Execution Process (EEDEP) support the implementation 
of the HEA. 

The Department’s future “to-be” architecture development includes the evolving 
Joint Technical Architecture (JTA), common Operating Environment (COE) and C4ISR 
Eramework supported by the Technical Reference Model (TRM) with a major focus to im¬ 
prove interoperability and information exchange. The COE, JTA, and C4ISR framework 
support the foundation of the software-intensive Department’s To-Be Architecture. 

The Department’s Modeling and Simulation Domain prescribed by [TRMOla] in¬ 
cludes the Common Technical Eramework (CTE) [DoD95, GMS+96] for facilitating M&S 
interoperability and reuse. The CTE has three components: the High-Eevel Architecture 
(HEA) to which the Department’s which M&S must conform; Conceptual Model of the 
Mission Space (CMMS) to provide a basis for the development of consistent and authorita¬ 
tive representations; and data standards to provide common representations of data across 
M&S and C4I systems. 
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Information technology standards are a means by which two or more products (or systems) can function together. [LibOl] 
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A conceptual model is the developer’s method of translating the M&S requirements 
into a detailed design framework to develop the software, and deseribes the conceptual 
model components, interaetions, and the M&S concept of operations. [SPC+98, PacOOa, 
SHOO, SheOOb, DDHOl] validated the basic need for eoneeptual models, and developed 
supporting perspectives of eoneeptual modeling theory, identified systemic issues, and pro¬ 
posed solutions. However, the Department’s M&S community currently lacks consensus 
and standards for eoneeptual models, which are eritical to the key Department’s M&S is¬ 
sues of authoritative representations, improved simulation eredibility, enhaneed confidence 
in simulation results, integrated simulation seeurity, and improved simulation interoperabil¬ 
ity. 

Surveying the eurrent M&S situation, [LRH+Ol] consider the term “conceptual 
model” extremely overloaded and overused within the M&S eommunity. For example, the 
term “conceptual model” deseribes the first abstraction of representations in an M&S and 
deseribes a high-level design of how all the components in an M&S relate to another. The 
wide and varied body of literature on the subjeet of conceptual models is an indication of 
their importance, and also reveals a lack of consensus on eoneeptual model methods, veri¬ 
fication and validation, format and development methodology. 

The term Functional Description of the Mission Spaee (FDMS) replaced the previ¬ 
ous Conceptual Model of the Mission Spaee (CMMS) term, in the Department’s M&S 
lexieon, as a required component of the M&S development and V&V processes. The 
FDMS provides the real-world deseription of entities, proeesses, and aetivities for the de¬ 
sign phase, and detailed representations of the problem domain in the requirements devel¬ 
opment phase of the simulation life cycle. 

Data standards are the third eomponent of the common technical framework for 
M&S. In any M&S application, the associated data quality should be as ereditable as the 
M&S itself if the user or sponsor is to attach confidence to the results. [DoD95] defines 
data quality for the Department’s M&S programs. [BCE-l-99] also suggests data quality 
attributes ineluding souree accuraey, fidelity, suitability, credibility, maturity, cost, avail¬ 
ability, and similar characteristics. [Kil02] recommends data aecuraey and quality informa- 
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tion supported by a review at three levels: database, data element, and data value; employ¬ 
ing descriptive, quality and usage metadata. 

Theoretically, the data producer uses metadata to describe the quality of the data as 
a part of data V&V process to meet the specification requirements and provides data meet¬ 
ing the user’s specification. The user assesses the applicability and quality of the data for a 
specific application based on the producer’s verified metadata. As part of the users V&V 
process, the data verification process checks the model algorithms, and as the last step, 
validates the data and algorithms to establish overall M&S credibility. 

Environment data is critical to the M&S community. However, quality environ¬ 
mental data is hard to acquire. In an early DMSO sponsored initiative to improve envi¬ 
ronmental representations, the Naval Research Laboratory developed the Environmental 
Effects for Distributed Interactive Simulation (E DIS) program. The Synthetic Environ¬ 
ment Data Representation and Interchange Specification (SEDRIS) program is a major De¬ 
partment to improve environment representations. The SEDRIS vision is to provide a 
common synthetic environment to reduce cost, improve reuse, support interoperability, and 
provide effective data transfer between heterogeneous systems, initially in the training 
functional area. The SEDRIS program employs a meta-model based transmittal methodol¬ 
ogy with a standardized data model and supporting application program interfaces. 

Complementing these initiatives are current or projected efforts to acquire science 
data from other the private sector, international scientific programs, or cooperative public- 
private partnerships. However, remotely sensed data from space may be difficult to share 
based on different sensor characteristics (e.g., sensitivity, spectral coverage, calibration), 
the lack of standards, proprietary data processing systems and algorithms, and subtle varia¬ 
tions that may adversely influence data consistency. In addition, ensuring commercial data 
sources are available requires an understanding of negotiations for intellectual property. 
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VI. MODEL QUALITY ATTRIBUTES FOR IMPROVED CREDIBILITY 

A. INTRODUCTION 

Chapter VI addresses software and simulation model quality attributes affecting 
heterogeneous system representation anomalies and credibility in Department M&S, espe¬ 
cially in federation interoperability. The chapter reviews the internal and external attributes 
of the ISO 9126-1 Quality model [ISOOl], testing for quality attributes, and M&S quality 
attributes addressed in much of the current literature, including an overview of aggregation 
and disaggregation. The chapter concludes with a discussion on M&S quality approaches. 

B, SYSTEMIC SIMULATION SOFTWARE QUALITY ISSUES 

In the 1970s, the cost of software maintenance for the first time exceeded the cost 
of software development, creating a “software crisis” and identifying for the first time that 
the technology at the time was inadequate to develop large, complex software-based sys¬ 
tems. By the mid-1980s “silver bullet” solutions such as computer-aided software engi¬ 
neering (CASE) tools were tried and found wanting to solve the “software crisis” as soft¬ 
ware evolved into a $300 billion dollar a year industry [RakOl]. 

In 1988, the Department’s Director of Operational Test and Evaluation (OT&E) 
[Kri88] acknowledged that the evolving art of M&S needed guidance to improve quality 
and establish credibility of simulation results through verification, validation and credibility 
assessments. [Kri89] implemented a policy on M&S support of OT&E specified by 
[DOT89] that identified the following high-level objectives in the model development 
process supporting credibility, improved quality and confidence in M&S results: 

• Acceptability of the M&S approach, 

• Confidence in the model, 

• Confidence in the M&S team, 

• Confidence in methodology and use, 

• M&S verification, validation and accreditation [DOT89]. 

Society’s growing dependency on large, complex software-intensive systems com¬ 
plicates the development of quality software. According to [BooOl], as “software contin¬ 
ues to weave itself deeply into the fabric of society, the stakes have gotten higher” [BooOl]. 
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This challenge has driven the development of many different methods, teehniques and ap- 
proaehes to improve software quality supporting the Department’s large-scale, software¬ 
intensive, simulation systems. 

C. SOFTWARE QUALITY INITIATIVES 

Today, Department testers still require suffieient eonfidenee in the quality of M&S 
results [DOT99, COHOO], to support a foundation of trust for using M&S results as the ba¬ 
sis of management deeisions or part of the formal test and evaluation process. The De¬ 
partment initiated several programs to improve the relationship between M&S and live test¬ 
ing, reduce costs and improve the overall testing proeess. In 1995, the Secretary of De¬ 
fense approved five initiatives, ineluding the more effeetive use of M&S, to mature the 
OT&E eommunity culture from a pass/fail, event-driven paradigm to the view of testing as 
a learning process. 

[Wai02b] discusses several of these initiatives, ineluding the role of hardware-in- 
the-loop M&S in missile defense systems testing and the emerging eollaborative testing 
environments developing in teehnical testing arehiteetures sueh as the Test and Training 
Enabling Architeeture (TENA) [PEE-l-02] and the Virtual Proving Ground (VPG). In an 
effort to improve M&S management in the Department’s test and evaluation proeess 
[Obr99] proposed the Modeling and Simulation Test and Evaluation Reform (MASTER) 
initiative, based on speeifie eategories of expertise (e.g., terrain, weather), ete ealled M&S 
vectors. The MASTER coneept [Obr99] also supported verifieation and validation of as¬ 
signed models performed by government R&D centers, and a consortium organizational 
structure to implement policy and maintain a repository of codes. 

Quality is an essential attribute supporting the safe operation of M&S software sys¬ 
tems, espeeially M&S involving possible life or death situations [Bow02]. [EevOO] de- 
seribed the root eauses found in the software industry’s flawed quality processes, whieh 
ereated the eonditions for the Ariane 5 failure in 1996 resulting in an uninsured five billion 
dollar loss. Several major spaceeraft losses ineluding NASA’s Mars Polar Eander, Mars 
Global Surveyor, and the Titan IV / MILSTAR spacecraft [EevOIa], revealed similar soft¬ 
ware engineering quality shortfalls. It is probable that similar software engineering quality 
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and safety issues exist within the Department’s software engineering programs today ae- 
eording to [HNC+00]. Furthermore, [HNC+00] found the current level of the Depart¬ 
ment’s software engineering practice lacks discipline and consistent enforcement mecha¬ 
nisms necessary to improve these conditions. 

There have been several uses and revisions of the term software quality including 
[MRW77, IEE89, IEE92, IEE98d] over the past twenty-five years. Today, software quality 
remains open to subjectivity, different views, and various interpretations of definitions. 
[MRW77] addressed software quality attributes in an early effort and proposed a software 
quality model that identified three categories of software quality attributes. The study of 
software quality and software quality metrics [IEE89, IEE98d] evolved continuously with 
the maturation of software engineering practices and the growth of the software industry. 

The ISO developed a two-part software quality specification model [ISOOl], pro¬ 
vided at Eigure 6-1, applicable to every type of software product. The current [ISOOl] 
definition of quality, similar to the IEEE definition cites “the totality of characteristics of an 
entity that bare on its ability to satisfy stated and implied needs” [ISOOl]. The ISO Quality 
Model identifies six characteristics for internal and external product quality and their asso¬ 
ciated sub characteristics. An additional four quality-in-use characteristics prescribed by 
[ISOOl] describes the combined effects of internal and external product quality. 

The study of software quality by [Dro95, Dro96] suggests that the emphasis on 
quality software development has been heavily process-oriented and proposes a product- 
based model composed of components and modules possessing three key quality proper¬ 
ties: cohesion, coupling, and layering^^"^, linked to quality attributes. As object-oriented 
(00) languages continue to evolve, [AleOl] also addressed the additional complexity of 
00 languages over previous procedural programming testing techniques in 00 testing, in¬ 
cluding inheritance, polymorphism and complex data requiring adequate testing of all rela¬ 
tionships. [EowOl] suggests the earliest indication of design quality is the level of cou¬ 
pling, high-level dependencies and cohesion. 


Software quality is the degree to which software possess a desired combination of quality attributes [IEE98d]. 
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Layering for objects, programs, processes and systems are governed by the constructive principle that components constructed at one 
level may only be used at the next higher level. [Dro96] 
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Figure 6-1. The ISO 9126-1 Quality Model (From [ISOOl]) 


[IEE98d] provides a standard for software quality metries supporting assessments 
for both the process and the product on the status of meeting quality requirements through¬ 
out the software life cycle. [Sch02a] incorporates quality metrics into software engineering 
practices, citing a growing quality measurement body of knowledge [lEEOOa] augmenting 
the IEEE software quality standard. [BD02] added an extension to the [Dro96] hierarchical 
methodology quality framework work of developing the Quality Model for Object-oriented 
Design (QMOOD) to improve object-oriented design quality. [BD02] based the QMOOD 
model on the following broad quality attributes: functionality, effectiveness, understand- 
ability, extendibility, reusability, and flexibility, in lieu of the ISO 9126 quality attributes 
[ISOOl] listed in Eigure 6-E 

Methodologies for improved software quality continue to appear. [BHKOl] sug¬ 
gests software documentation inspections in the early stages of a software project may al¬ 
low for the early identification of defects, and experiments conducted by [BHKOl] support 
the employment of a second inspection cycle to improve overall quality. [HJB-l-98] took a 
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different approaeh and developed a method for developing produet/proeess dependeney 
models (PPDMs), empirieal software artifaets, refleeting past experienees to improve the 
planning, teehniques, tools or methods for future projeets. In addition, [FPC97, KisOl] 
identified measurement models and frameworks by supporting the software engineer’s abil¬ 
ity to establish produet measures supporting improved overall software quality and reliabil¬ 
ity. [BooOl] suggests properly applied measures provide management with important in¬ 
sights into the health and welfare of a software projeet. 

1. Internal / External Quality 

[ISOOl] divides software quality into six major oharaeteristies: funetionality, reli¬ 
ability, usability, efficieney, maintainability, and portability, with eaeh oharaeteristie eom- 
posed of sub-charaeteristies . Funetionality is the eapability of the software produet 
funetions to meet stated and implied needs under eertain speeified eonditions, and ineludes 
the sub-eharaeteristics of suitability , aeeuraey, interoperability, and seeurity. 

The [DoD98] glossary defines aeeuraey as “the degree of exaetness of a model or 
simulation, high aeeuraey implying low error” . In addition, “aeouraey equates to the 
quality of the result, and is distinguished from preeision” [DoD98]. A signifieantly differ¬ 
ent definition of accuracy by [RGHOO] cites accuracy as, “the agreement between the per¬ 
formance of these models of each aspect and the real world performance” [RGHOO]. An 
additional definition from the [RPGOO] guidelines defines accuracy as the “degree parame¬ 
ters, parameter sets, or variables correspond exactly to the simulation reality, referent or 
some chosen standard” [RPGOO]. 

Accuracy is also one of several model attributes and dimensions related to model 
real-world fidelity, and a quality sub-characteristic of software functionality identified by 
[ISOOl] as the capability to achieve the agreed upon or correct solutions with the required 


There are compliance sub-characteristics for all six characteristics with similar standards and protocols [ISOOl]. 

Suitability is the software’s fitness of purpose to provide an appropriate set of functions to complete certain tasks, meet user objec¬ 
tives, and affects operability [ISOOl]. 
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Error: The difference between an observed, measured, or calculated value and a correct value [RPGOO]. 
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degree of precision^^^. Aceording to [Kil02] eonfidence in software quality and aceuracy 
depends on the development environment, verification evidence, and software quality as¬ 
surance [IEE89] and assessments, which includes accepted software development practices, 
personnel skill and experience attributes, sufficient resources and the existence of key life 
cycle development artifacts such as configuration management histories and logs. [Kil02] 
also notes, “dedicated software engineering experience is essential to most large-scale 
software development efforts” [Kil02]. 

M&S interoperability is also an extremely complex systemic issue, and one of the 
central aspects of the Department’s current M&S efforts. Major M&S interoperability is¬ 
sues confronting joint model development and federations include data, algorithms 
[CER97], representations, and joint C3I [You97, DPB+Ol]. [ISOOl] defines interoperabil¬ 
ity as the ability of the software to interact with specified systems. However, interoperabil¬ 
ity is not just an M&S specific challenge, it is a universal Department issue, which impacts 
many sectors of the Department information domain [Ngu95, Wof95, HKE96, AH97, 
HMD99, Sut99, DSB+99, HamOOb, HDGOO, YBG+01, HMOO, GBEOl, EBG+01, 
WHE+01, BKW+02, CW02]. Software developers and maintainers may introduce inter¬ 
operability anomalies in autonomously developed software-intensive systems in all phases 
of a systems life cycle. 

Interoperability is also a multi-dimensioned challenge. The interoperability re¬ 
quirements listed by [DoD02d] are similar to the architecture concept for seamless M&S 
with a single integrated environment proposed by [Dow92]. The synthetic environment 
ideas advanced by [Dow92] identifies three classes of requirements: 1) simulation truth, 2) 
conceptual consistency, and 3) temporal consistency, which generate the supporting techni¬ 
cal dimensions including the a) system architecture, b) data management, c) human ma¬ 
chine interaction, and d) time management. [Ham96, HNP97] and [HM02] address the 
management of time, clocks, and synchronization in distributed simulation, the key role of 


Precision relates to the quality of the operation by which the result is obtained and can be repeated, and is distinguished from the term 

accuracy that is the quality of the result. [DoD98] 
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M&S interoperability is the ability of a model or M&S to provide services to and accept services from other M&S, and to use the 
services so exchanged to operate effectively together [DoD98] 
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partitioning, and identifies five methods for partitioning a simulation for distributed exeeu- 
tion, with synehronization noted as the most important implementation eonsideration. 

The Internet is a major new interoperability dimension. With the explosive growth 
in the Internet in the I990’s, the Clinton Administration eoneeived a National Information 
Infrastrueture (Nil) program to deliver to all Amerieans the information they needed at an 
affordable priee. A eomplementary Department initiative effort to the Nil, the Defense In¬ 
formation Infrastrueture (DII) [TM97], provides support for “dual-use” teehnology and 
added privaey, reliability, and seeurity quality attributes to Nil initiatives. The DII founda¬ 
tion elements^"^'^ [TM97] ineluded modeling and simulation, standards, arehiteetures, soft¬ 
ware engineering, and test and evaluation. 

However, [SFJ+98] eited eontinuing interoperability issues at almost every level. 
The Department determined a need to develop better polieies and proeedures to align and 
resouree the teehnology base to support information assuranee [DoD02e], information dis¬ 
semination management, interoperability, network management, network operations, and 
enterprise eomputing and implemented the Global Information Grid^"^^ (GIG) and the GIG 
Arehiteeture^"^^ (GIGA) [GIGOO]. [HamOOb] provided Departmental GIG poliey and guid- 
anee. [DPB+OI, ChaOI, WolOI, BBN+OI] also identified the need for new analytieal tools 
to support improved interoperability and faeilitate the industrial age to information age 
transformation proeess. 

The seeurity quality attribute^"^^ addresses the ability of the software to prevent un¬ 
authorized aeeess to programs or data, whether aeeidental or deliberate [ISOOl]. The Na¬ 
tional and the Departmental strategies to eounter threats and seeurity ehallenges 
[WHM+79, LX99, KH02, SB02a] inelude natural environment, man-made physieal haz¬ 
ards, and human aetors, whieh are now eonsidered more eomplex and more diffieult than 
earlier threats [WHM+79]. The eomplex global interdependent information environment 
identified by the worldwide Year 2000 remediation effort [Gre97, WG99, Mus02] ineludes 

DII foundation elements included policy, requirements, modeling and M&S, standards, architectures, technology base, software engi¬ 
neering, test and evaluation, and joint spectrum management [TM97]. 
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The GIG is “a globally interconnected end-to-end set of information capabilities, associated processes and personnel for collecting, 

storing, disseminating, and managing information on demand to warfighters, policy makers, and support personnel” [HamOO, GIGOO]. 
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The GIGA is composed of interrelated operational, systems, and technical views, which defines the characteristics of and relation¬ 
ships among current and planned Global Information Grid assets in support of National Security missions [GIGOO]. 
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Security quality attributes are: availability, identification, authentication, confidentiality, integrity, and non-repudiation [WooOOb]. 
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critical infrastructureprotection [EFL+97, ABBOO, BusOl, GPH+01], information opera¬ 
tions (10)^"^^ doctrine [JP98], 10 opportunities and vulnerabilities addressed by [SBJ+02], 
and information superiority issues [JP98]. 

The Department is just beginning to understand the risks to an information superior¬ 
ity strategy and information operations (10) as a basis for safeguarding the Nation’s critical 
interests, and responding with information assurance (lA) measures. Information assur¬ 
ance'"^^ includes risks and protective measures [HNP97, HamOOb, ABBOO, WooOO, 
MonOOb, WFP+01, DoD02d, DoD02e]. The Department selected Millennium Challenge 
02 (MC02), a large-scale experiment to define the impact of 10 measures employing the 
Joint Staff Analysis Model, as part of a limited-objective lO/IA experiment [A1102]. Initial 
MC02 results suggest that modeling advanced 10, IS, and lA eapabilities present signifi¬ 
cant challenges to the Departmental M&S eommunity. 

Reliability, the second of six major [ISOOl] software quality characteristies, in¬ 
volves the capability of software used under certain conditions to meet a specified level of 
performance. Reliability is comprised of the following sub-characteristies: maturity, fault 
tolerance, and recoverability [ISOOl]. Since software does not wear out or age, limitations 
in reliability are rooted in the requirements, design, implementation, and maintenanee 

processes. One hundred percent software reliability may be an impossible goal to attain in 

1 

the eurrent software development environment when one considers that even ‘six-sigma’ 
strategies seek only to reduce the number of errors in software systems. 

Software reliability engineering (SRF)*"'^ practices are critical in high-risk, safety- 
critical areas, whieh affeet national security or risk human life. NASA Shuttle mission en¬ 
gineers successfully implemented the Space Shuttle Flight Software Application methodol- 
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Critical infrastructure was defined in the 1998 Presidential Decision Directive 63 as “those physical and cyber-based information 

systems essential to the minimum operations of the economy and the government” [SBJ+02]. 
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Information operations are operations conducted to defend our own information and information systems and affect adversary infor¬ 
mation and information systems [WooOObj. 
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Information superiority is the capability to collect, process and disseminate an uninterrupted flow of information, while exploiting or 

denying an adversary’s capability to do the same [WooOObj. 
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Information assurance, a subset of information operations, is actions that protect and defend information and information systems by 
ensuring availability, integrity, authentication, confidentiality, and non-repudiation, and includes restoration of information systems by 
incorporating protection, detection, and reaction capabilities [WooOObj. 
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Six Sigma objectives are to reduce the number of defects to 3.4 defects per million lines of code [STSOOj. 
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SRE is the application of statistical techniques to data collected during system development and operation to specify, predict, esti¬ 
mate, and assess the reliability of software-based systems. [Sch99aj 
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ogy [KS97, Sch97a, Sch97b, Sch99a] to improve overall Shuttle software reliability. 
Software reliability affects all aspects of the software life cycle to include architecture 
[Lew02a], requirements [SchOla, SchOlb], risk [SchOld], software maintenance process 
[Sch99b], commercial-off-the-shelf software use [SchOOc], client-server systems [Sch96], 
testing [JDLOl], and software quality control and prediction [SchOOc]. [Rus98, IC99, 
RC99, ICOl, RCOl] describe a decision support tool for selecting a reliability strategy 
based on project, product and resource decision factors. Other software reliability sub¬ 
characteristics include maturity, fault tolerance, and recoverability where: 

• Maturity is the ability of the software to avoid failure stemming from faults in the 
software product, or the frequency of failures, 

• Fault tolerance is the robustness of the system and its ability to maintain a specific 
level of performance in the event of faults or specified interface problems, and may 
include a fail-safe capability, 

• Recoverability is the software’s ability to regain a specified level of performance 
and recover affected data in the event of a failure [ISOOl]. 

Usability is the third of six major [ISOOl] software quality characteristics. Usabil¬ 
ity includes the ability of a user to understand, learn, and employ the software produce un¬ 
der specified conditions and is comprised of the following sub-characteristics: understand¬ 
ing, leamability, operability, and attractiveness [ISOOl] where: 

• Understandability is dependent on the documentation and the initial impression of 
the software, and allows the user to determine if the software is suitable, and how it 
may be used for specific tasks and under what conditions, 

• Leamability is the ability of the user to learn to how to use the application, 

• Operability is the ability of the user to operate and control the software product, and 
is affected by the attributes of suitability, changeability, adaptability, and installabil- 

ity, 

• Attractiveness is normally associated with the graphical design, uses of color, and 
the ability of the software to be attractive to the user [ISOOl]. 

The usability sub-characteristics are subjective, dependent, most difficult to meas¬ 
ure, and are the least objective quality factors. [JWCOl] acknowledge that usability is a 
difficult characteristic to integrate into any system, and requires specific knowledge of user 
requirements, preferences, and limitations. Research by [GRS02] borrowed from the psy- 
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chology field in studying the relationship between pain researeh and the implieations for 
usability in software engineering, speeifieally the peak-end effeet.'^® 

Research of simulation software usability by [Kil02] suggests that simulation soft¬ 
ware is credible only when employed within a well-defined context of use; that usability is 
more a case of reducing the probability of simulation misuse rather than the ease of use; 
and includes the M&S user support activities, which facilitate the credible use of the M&S. 
[DoDOla] found most Department simulations were too difficult to use and too resource 
intensive to employ as widely as needed to achieve the complete range and scope of re¬ 
quired objectives. [Kil02] also identified configuration management as an indicator of 
M&S usability and the “glue that ties the version of the simulation to all the V&V results 
and M&S documentation” [Kil02]. 

Efficiency, the fourth of the six major [ISOOl] software quality characteristics, in¬ 
volves the software’s resource dependent ability to provide appropriate performance, under 
stated conditions. Time-behavior and resource utilization comprise the [ISOOl] sub¬ 
characteristics of efficiency where: 

• Time behavior allows the software under stated conditions to provide appropriate 
responses, throughput rates, and processing times, 

• Resource utilization is the ability of the software, under prescribed conditions to 
perform its function using appropriate amounts and types of software [ISOOl]. 

The fifth of six major [ISOOl] software quality characteristics is maintainability. 
Maintainability addresses the ability to modify the software product, including corrections, 
improvements, or adaptation of the software to changes in the requirements, environment 
or functional specifications, and includes the following [ISOOl] sub-characteristics where: 

• Analyzability is the ability to diagnose the software for deficiencies, failures, or for 
ease of maintenance and modification,. 

• Changeability incorporates design, coding, and documentation and the ability to 
implement a specific modification, 

• Stability allows the software to overcome unintended effects from software modifi¬ 
cations, 

• Testability provides the capability to validate the modified software [ISOOl]. 


Redelmeier and Kahneman discovered the peak-and-end effect in patient study research when patients were asked to rate their pain at 
regular intervals during the procedure [GRS02]. 
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Portability, the sixth and last of the six major [ISOOl] software quality eharaeteris- 
tics deals with the sub-eharaeteristies of adaptability, installability, eo-existenee, and re- 
plaeeability to support standards and eonventions of software portability from one envi¬ 
ronment to another. 

2. Quality in Use 

The [ISOOl] standard speeifies quality-in-use eharaeteristics as enablers allowing 
users to satisfaetorily, effectively, productively, and safely, achieve specified objectives. 
The [ISOOl] quality-in-use characteristics include effectiveness and productivity where: 

• Effectiveness is the software’s ability to accurately and completely achieve speci¬ 
fied goals, 

• Productivity is the software’s ability to achieve results effectively when compared 
to the expended resources [ISOOl]. 

Technical constraints include the lack of tools, data security, data descriptions, variable 
resolutions [HOB95], and hardware/software limitations. Cultural challenges include the 
lack of trained personnel, immature processes and the lack of acceptance of M&S tools. 

Safety is the software’s ability to maintain acceptable levels of risk to people, envi¬ 
ronment, property, business, or software [ISOOl]. [LevOO, LevOla, LevOlb, LevOlc] iden¬ 
tified the causal factors related to software safety issues in recent air and space accidents 
including the 1996 explosion of the Ariane 5 launcher, the 1999 loss of the Mars Climate 
Orbiter, the placement of a MILSTAR satellite in an incorrect and unusable orbit in 1999, 
and the destruction of the Mars Polar Lander in 2000. Several additional space accident 
reports [LLF-i-96, SMB-i-99, YAB+OO] and aircraft accident reports [Lad93, SL94, Lad95] 
revealed very similar systemic safety issue factors including, but not limited to: 

• Overconfidence and over-reliance on digital automation, where software complex¬ 
ity is underestimated and the effectiveness of testing is overestimated, 

• Not understanding the risks associated with software, 

• Confusing reliability with safety, 

• Over-reliance on redundancy, 

• Assuming risks decrease over time, 

• Ignoring warning signals, 

• Inadequate cognitive engineering, 

• Inadequate specifications. 
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• Flawed review proeesses, 

• Inadequate system safety engineering, 

• Violation of basie safety engineering praetiees in the digital parts of the system, 

• Software reuse without appropriate safety analysis, 

• Unneeessary eomplexity and software funetions, 

• Operational personnel not understanding automation, 

• Test and simulation environments that do not match the operational environment, 

• Deficiencies in safety-related information collection and uses [LevOlb]. 

[LevOla] expanded upon the cited accident-causal factors and noted their existence 
in three systemic organizational categories including flaws in the safety culture, ineffective 
organizational structure and communication, and inadequate or ineffective technical activi¬ 
ties. In aerospace systems, a general testing heuristic is to fly what you test and test what 
you fly. However, [LevOlb] noted in the Ariane 5 case that developers experienced all 
three accident-causal factors and violated the general testing heuristic when they reused the 
trajectory data of the Ariane 4 in the simulation and specification of the Ariane 5 software, 
even though the trajectories were different. 

[Lev95, HNP97] also cited similar software safety issues as the cause of the Therac- 
25 computer-controlled radiation therapy machine accidents, which caused massive radia¬ 
tion overdoses in six people between June 1985 and January 1987. [WLL+01] introduced a 
hierarchical accident model to help identify safety factors considerations with three levels 
of abstraction: 1) Level I factors include the chain of events; 2) Level II factors included 
the conditions or lack of conditions allowing the events at Level I to occur; and 3) Level III 
factors include the root cause or systemic factors of an accident. 

Satisfaction, the final [ISOOl] quality-in-use characteristic includes the software’s 
ability to satisfy users. There are also a myriad of issues the Department must address to 
achieve customer satisfaction and confidence including: 

• Credibility in Department M&S, 

• Improved Department decision-makers confidence in simulation results, 

• A quantifiable return-on-investment (ROI) methodology for simulations, 

• Additional validated automated processes and tools for models and simulations, 

• Simulation utility keeping pace with user requirements and decision-makers expec¬ 
tations [SchOlc], 

• Interoperability of legacy M&S systems with war fighting systems [DoDOla]. 
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3, Testing for Quality Attributes 


[BM93] identified four distinet periods of software development identifiable by the 
following dominant themes and goals: 1) the funetional era (1960s), 2) the sehedule era 
(1970s), 3) the eost era (1980s), and most reeently, 4) the quality era (1990s). Software 
testing [DPR+94, Roa98, ClaOl, PLPOl, RakOl] is a eritieal aspeet of quality analysis and 
is most effective when based on quantitative measures integrated in the development envi¬ 
ronment, and supporting the test management process. Software testing is a complex un¬ 
dertaking, which addresses many dimensions including new test challenges for compo¬ 
nents, Java, active agents, object-oriented technology, software reliability engineering 
[IC99, ICOl], website testing, test management, and the human element. Additional soft¬ 
ware test activities and challenges include the employment of Bayesian graphical models 
[WGC02], the context of software testing in the development process, the different roles of 
testers and developers [WhiOO], statistical software engineering [JA97], and test case pri¬ 
oritization to increase the rate of fault detection [EMR02]. 

Software testing is an essential component of simulation software development, a 
critical technique for evaluating product quality, and essential for improving product qual¬ 
ity by identifying defects and problems early in the development process [Som95, Pre97, 
Roa98, RakOl]. Software companies face major challenges testing products and [WhiOO] 
predict these challenges will continue to grow with the increasing complexity levels of the 
software. [RTI02] cited improved software testing*^' as a critical factor supporting soft¬ 
ware quality and reducing the software errors, estimated to cost the U.S economy $59.5 
billion annually. 

After recent mission failures, NASA developed an independent verification and 
validation (IV&V) implementation approach [RosOl] for all agency software development 
with a focus on high-risk cutting edge projects. NASA based the new IV&V approach on 
nine factors, which NASA determined impacted software development including: software 
team complexity, contractor support, organization complexity, schedule pressure, process 


Software testing is the process of applying metrics to determine product quality and is the dynamic execution of software and the 
comparison of the results against a set of pre-determined criteria. [RTI02] 
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maturity of software provider, degree of innovation, level of integration, requirements ma¬ 
turity, and software lines of code, evaluated against five risk categories [RosOl]. 

Software testing is also a best practice process and technique in the Department’s 
developmental test and evaluation procedures according to [SNF+99]. Although they share 
similar test techniques and methods, [BCC+Olb] differentiates software testing from soft¬ 
ware maintenance, noting that software testing is a component of the development process, 
whereas, software maintenance addresses system failures after the software is delivered. 
Software testing may also include static verification techniques. Software testers normally 
conduct testing at the unit, integration, and system level, applying a number of techniques 
including black-box and white-box testing, supporting different objectives cited by [Roa98, 
RakOl]; 

• Acceptance / Qualification testing, 

• Installation testing, 

• Alpha and Beta testing, 

• Conformance / Functional / Correctness testing, 

• Reliability / Achievement / Evaluation testing, 

• Regression testing, 

• Performance testing, 

• Stress testing, 

• Back-to-back testing, 

• Recovery testing, 

• Configuration testing, 

• Usability tests [Roa98, RakOl]. 

New verification techniques continue to evolve and now include the time warp 
technique for synchronizing parallel discrete event simulators [FRC+02]; model-based 
verification [GB99]; model extraction techniques for automated verification [HS02]; assur¬ 
ance-based testing [Pau99]; compositional verification methods [WH02]; and monitors 
employed as oracles or supervisors to analyze target systems [PP02]. [DMSOOb] proposed 
that component-based M&S methodology articulated by [HusOO] “could allow more effec¬ 
tive and affordable M&S verification” [HusOO]. 
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4, Model and Simulation Quality Attributes 

The M&S literature eontains a wide range of M&S quality dimensions and attrib¬ 
utes supporting key M&S eredibility eoneepts, ineluding the terms fidelity and aeeuracy. 

In praetiee today, the Department’s M&S eommunity often use the terms fidelity and aeeu- 
raey synonymously. Unfortunately, the Department’s M&S eommunity laeks consensus on 
a fidelity definition, how to measure it, what it costs, or even its relative importance for 
achieving confidence and credibility in M&S results. The casual use of other terms closely 
related to fidelity such as resolution, detail, aggregation, and multi-resolution add to the 
general confusion limiting the common understanding of fidelity. This adversely affects 
simulation credibility and confidence in the simulation results. 

a. Fidelity 

Although fidelity is a critical component for credible M&S, it has proven 
elusive to implement and resisted multiple theoretical efforts [Bai92, Ham96, GF97, FY97, 
Pac97, SFL+97, HNP97, Fay98a, Fay98b, GFT98, GZ98, Had98, Har98b, McD98, 

Pac98a, Pac98b, RGJ98, SBL98, Pea99, Har99b, MBH99, RVJ+99, GJOO, LMR+00, 
PacOlb, HYOla, HYOlb, BP02, RIO02] to describe in objective or quantitative terms. In 
addition, [HNP97] suggest bounding fidelity, defining perfect fidelity as a simulation indis¬ 
tinguishable from reality, which may be possible for some systems (e.g., virtual reality), 
although still out of the theoretical realm for non-trivial simulations. 

The [DoD95] addressed the issues of fidelity in three of the six major objec¬ 
tives (e.g., objectives two, three, and four). The ability to describe and quantify M&S fi¬ 
delity or “goodness” appropriately is essential for fully achieving other objectives identi¬ 
fied in the master plan, such as HLA (Sub-Objective 1-1), CMMS (Sub-Objective 1-2), and 
support of endeavors such as Simulation Based Training, Analysis, and Acquisition (Sub- 
Objective 5-1). Systemic issues confronting improved fidelity include challenges with 
physical, visual, audio, motion, and temporal, environmental and behavioral fidelity 
[DoD95]. 
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The [DoD95, DoD98] references contain the Department’s approved defini- 
tion for fidelity . According to the [RPGOO], fidelity is a core concept involved in every 
issue in M&S, especially issues related to V&V. Fidelity is also one of the key areas di¬ 
rectly affecting the credibility of the Department’s M&S programs and the confidence in 
the simulation or federation results provided to decision-makers. Fidelity is inherently an 
abstraction or a representation of some component of reality, a simuland , or referent de¬ 
veloped in M&S. Table 6-2 provides an overview of the fidelity context from the 
[RPGOO]. 

However, in practice, the M&S community describes fidelity by a number 
of dimensions and attributes: - accuracy, resolution, aggregation, de-aggregation, detail, 
extent, granularity, precision, repeatability, time, and spatial consistency. Many of these 
terms are ill defined and often cited interchangeably in the current literature. [SFL-l-97] de¬ 
fine “fidelity” as a measure of how accurately and realistically a simulator or simulation 
represents the “real” world. [GFT98] provide a definition for fidelity noting “the extent to 
which the model reproduces the referent, along one or more aspects of interests” [GFT98]. 
[RGHOO] provide two additional definitions of fidelity with the caveat that fidelity may de¬ 
scribe model representations, a simulation, simulation data, or an exercise, with different 
implications for each use. [Ham96, HNP97] addressed the benefits of model simplifica¬ 
tion^^"^ and decomposition^^^, without a loss of fidelity, especially of non-trivial systems. 

There are many more definitions of fidelity in common use today, exacer¬ 
bating the issue of defining fidelity. By 1992 there were at least twenty-two different defi¬ 
nitions of the term fidelity cited by [LA92] for distributed interactive simulations, further 
subdivided into twenty different hardware and software components, requiring different 
levels of fidelity. The heuristic practice for describing fidelity includes short imprecise, 
subjective, qualitative descriptions for fidelity such as low, medium or high. In addition to 
these “short descriptions” for fidelity, [RGHOO] identifies “shorthand descriptions” such as 
the Level D flight simulation classification consisting of over 100 attributes used by the 
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Fidelity is specified as the accuracy of the representation when compared to the real world [DoD95, DoD98] 
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The simuland is the representation the simulation is intended to represent, not necessarily the real world, with the referent being the 

sum total of what is known, assumed, or projected about the simuland [RPGOO]. 
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Model simplification includes three classes: brevity, clarity, and efficiency [HNP97]. 

An alternative strategy to simplification, decomposition breaks the system down into component parts to model [HNP97]. 
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Federal Aviation Administration, and “long deseriptions” of multiple attributes involving 
enumeration, which requires quality attributes such as accuracy, resolution, or both. 

[Gro99, RGHOO] characterizes past approaches to specifying or measuring fidelity as ad 
hoc, artistic, or problem / domain specific. 

[GFT98] propose that fidelity is the key to simulation validation, however, 
[Gro99] states, “fidelity has to be the least consistently used, yet most commonly used term 
in the M&S community” [Gro99]. [Mey98] concurs with [Gro99] and considers the term 
fidelity overloaded, with no context-free meaning, and almost useless in addressing the 
goodness of a simulation or federation. [PacOlb] adds that no widely accepted set of terms 
yet exists for the definitions of uncertainty, error, and variability as they apply to M&S fi¬ 
delity, validity, and their referents. 

[GF97] proposed that: 1) fidelity requires different measurements since there are 
fundamentally different aspects of fidelity; 2) fidelity is a property of simulation models; 
and 3) fidelity exhibits certain properties. Later, [GFT98, Pac98b] suggested the way to 
improve decision-makers credibility in the simulation and confidence in the simulation re¬ 
sults is to describe fidelity and accuracy quantitatively. However, [Gro99, RGHOO] con¬ 
tends there are two major obstacles blocking the development of a fidelity measurement 
standard: 1) there must exist an accepted definition of the real or imagined world with suf¬ 
ficient quantifiable characteristics to measure the difference between it and the simulation; 
and 2) the simulation must be similarly defined. 

Fidelity according to [Mey98] addresses the M&S measure of agreement 
with the perceived reality within a specific context and suggests that fidelity is relative to 
simulation resolution, appropriate for determining differences between the resolution of the 
simulation and the details and accuracy of the models. A major cooperative govern¬ 
ment/industry/academia consortium collaborated on DIS interoperability standards for 
IEEE approval from 1989-1995. The objective of the collaborative initiative P1278.5 enti¬ 
tled Standard for Distributed Interactive Simulation-Fidelity Description Requirements 
[IEE95, STR96] was the interconnection of dissimilar components of DIS to provide real¬ 
time applications with appropriate fidelity. Shelved in 1995, the [IEE95] failed to span the 
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problem space, generated insufficient support competing against the emerging HLA stan¬ 
dard, and lacked community support for its approach to fidelity. 

The simulation literature has a deep, diverse body of theory on fidelity and 
its impact on M&S credibility. [Pac98] reviewed basic fidelity concepts, definitions and 
theory, noting the need to be concerned about M&S fidelity. [Pac98] also provides a syn¬ 
opsis of fidelity ideas and issues providing a framework for dealing with fidelity through¬ 
out the M&S development life cycle. In addition, [Pac98] suggests methods to measure, 
estimate or quantify fidelity including adopting a methodology for deriving and publishing 
M&S fidelity metrics; economic issues; and other challenges which have plagued analytical 
approaches, including classical and Bayesian statistics, game and decision theory. How¬ 
ever, [Mey98] provides a counterpoint, arguing that quantifying M&S fidelity would be 
extremely questionable since M&S fidelity has almost no context-free definition. 

[Hug97b] further proposes that new technologies support 1) increased physical fidelity, 2) 
improved reasoning fidelity, 3) enhanced behavioral fidelity, 4) better flexibility, 5) im¬ 
proved information transfer and fusion. 

The failure to describe M&S fidelity appropriately may adversely influence 
current and future Department interoperability, transformation, business, and warfighting 
initiatives. [PacOlb] proposes a referent^^^ for fidelity and validity, where the fidelity ref¬ 
erent provides the system theory, and conceptually is the most complete collection of in¬ 
formation about the subject represented in the simulation, while the validity referent is 
more complex because it must involve intended use for the simulation. The referent is a 
standard from which the thing being represented in M&S is derived or the standard against 
which the correctness of M&S representation is measured [PacOlb]. [PacOOa] also dis¬ 
cusses approaches for a conceptual model, supporting enhanced model completeness, con¬ 
sistency, and coherency. 

[Had98] considers fidelity as an open issue, with respect to fidelity and reso¬ 
lution, scope, and completeness from the aspect of M&S developer. Moreover, [GF97] and 
[GFT98] note that M&S fidelity is a critical measure of M&S credibility, and one the larg¬ 
est cost drivers for M&S, although [GFT98] believes that fidelity is currently of little use 

A referent is a codified body of knowledge about a thing being simulated [PacOlb]. 
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when engineering the M&S requirements. In addition, [Pac97, Pae98a] notes that there is 
no specifie guidance for fidelity and that the M&S community has not yet agreed upon ba¬ 
sic fidelity concepts and definitions. [PacOlb] also provides a broad overview of the issues 
confronting improved simulation and federation credibility and confidence in the results, 
including the important topic of quantifying the validity of HLA federates and federations. 

In addition, [Had98] describes the problem of defining an appropriate level 
of fidelity for Conceptual Models of the Mission Space, and notes fidelity is the most diffi¬ 
cult characteristic to define because it applies to many aspects of a problem. For example, 
a simulation may contain a “high fidelity” with respect to a sensor representation, but still 
labeled as “low fidelity” with respect to other representations such as human behavior or 
threat. [RGHOO] added that fidelity requirements, specifications, and conceptual models 
have different research approaches although they share many shortcomings, including the 
lack of a formal and fundamental theory or framework. The [RPGOO] guide provides a fi¬ 
delity conceptual framework in Figure 6-2 with fidelity as a core concept integral to all as¬ 
pects of M&S, especially VV&A; comprised of attributes such as resolution, er- 
ror/accuracy, sensitivity , and precision; and concluding that qualitative terms inade¬ 
quately express fidelity. 

During the decade of the 1990’s, the Department and M&S community in¬ 
vested significant time, resources and attention to improve M&S verification, validation, 
and accreditation (VV&A). These W&A efforts established processes and identified 
VV&A responsibilities, but have not provided methods for quantifying fidelity require¬ 
ments or for determining complex M&S accuracy attributes quantitatively. Other groups, 
especially those involved in the Department of Energy’s Advanced Strategic Computing 
Initiative (ASCI) and others concerned with M&S of basic physical processes, have started 
significant efforts to quantify M&S validity [PacOlb]. These groups have begun to address 
questions of uncertainty and error for both M&S and experimental data used with the M&S 
as inputs and as standards for validity comparisons. 


Sensitivity; The ability of a component, model or simulation to respond to a low level stimulus [RPGOO]. 
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Figure 6-2. Overview of Fidelity Context (From [RPGOO]) 
b. Accuracy 

Accuracy, a sub-characteristic of the [ISOOl] quality characteristic dis¬ 
cussed earlier also plays a major role as an M&S quality attribute. Department M&S users 
also demand more accurate'elements of the mission space and developing authoritative 
representations remains a major objective of the draft Department M&S Master Plan, 
[DoDOla] currently under development. This has proved challenging to the Department 
since there are many views of accuracy. For instance, [Mey98] excludes the term simula¬ 
tion in his definition and proposes that accuracy “is a measure of the exactness of the 
model with respect to the characteristics and behaviors of the physical entity which the 
model represents” [Mey98]. 


Accuracy is 1) the degree of exactness of a model or simulation, high accuracy implying low error. Accuracy equates to the quality 
of the result, and is distinguished from precision, which relates to the quality of the operation by which the result is obtained and may be 
repeated [DoD98], 2) the degree to which a parameter or variable or set of parameters or variables within a model or simulation conform 
exactly to reality to reality or to some chosen standard or referent (SISO) [RPGOO]. 
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Providing another viewpoint, [Bal98] addressed the signifieant aeeuracy 
quality eharaeteristies inherent in verifieation, validation, testing, aeereditation, eertifiea- 
tion, and credibility assessment activities. In yet another view for the perspective of the 
data, [LevOO, Kil02] propose: 

• That data accuracy requires the appropriate data with the correct resolution for the 
intended use, 

• The proper quality of data established by the data producer and reviewed by the 
user, 

• The correct data transformation activities to make the data compatible with M&S 
[LevOO, Kil02]. 

There are also many other views of accuracy. [PacOOa] addresses conceptual model 
decomposition, and how the characteristics of the simulation elements are abstracted de¬ 
termine the accuracy and precision. [FY97] defines accuracy as being inversely propor¬ 
tional to an error measurement in which more accuracy implies less error. [FY97] also dis¬ 
cusses how to determine and trace M&S accuracy through the models and processes asso¬ 
ciated with each layer, ensuring that the accuracy meets the design requirements, ensuring 
the M&S operates within the maximum error bounds and thus meet its intended require¬ 
ments. 

[Gro99, RGHOO] suggest accuracy is determined by how well the simulation algo¬ 
rithm represents the subject that is simulated, and may be measured against reality, termed 
real accuracy^^^; or against the articulation of the an application domain or mission space; 
abstraction accuracy*^®; and should to relate to a single parameter or set of parameters. 

With the intent of achieving software engineering consistency, in this research, we employ 
the IEEE definition for accuracy provided in the HLA Framework and Rules [lEEOOc]. 
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Real accuracy is a function of both the correctness of the abstraction and of the simulation’s representation [Gro99, RGHOO]. 
Abstraction accuracy refers to the only the function of the simulation’s representation of the abstraction [Gro99, RGHOO]. 

Accuracy is the measure of the maximum deviation of an attribute or parameter value in simulation or federation from reality or some 
other chosen standard or referent [lEEOOc]. 
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c. 


Precision 


Precision is also a close relative to accuracy and often used as a synonym in 
the body of research. [Gro99, RGHOO] suggests that precision'^^ is a measure of the reso¬ 
lution or granularity with which a parameter may be determined, and always limits accu- 
racy^^^. Precision*^"^ receives additional clarification in the [RPGOO]. Preferring quantita¬ 
tive attributes to qualitative identifiers, the selected IEEE standard glossary of software en¬ 
gineering terms [IEE90] definition for precision^^^ suggests improved correctness in 
verification and validation processes, inferring higher credibility in the simulation and 
improved confidence in the results. 

d. Timeliness 

Timelinessa factor in considering simulation fidelity, cautions [Gro99, 
RGHOO] has the potential to create issues for parameter accuracy and precision in the parts 
of the simulation or federations, which may advance faster or slower than other parts. 
[Gro99, RGHOO] indicates these manifestations may occur in distributed simulations, in 
unitary simulations employing distributed processing techniques, or in discrete event simu- 
lations, and is dependent on time management techniques in the simulation . 

e. Error 
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Error is a significant M&S quality attribute. [Ham96, HNP97] note an er¬ 
ror occurs when a fault (e.g., a physical defect or flaw occurring in either the hardware or 


Precision may normally be determined in a simulation by an understanding of the systems computational processes such as the num¬ 
ber of significant digits, round off, interpolation intervals, update/refresh rates, and the quality of the real world data [Gro99, RGHOO]. 

Accuracy, precision, and timeliness characteristics describe how close the representation of an individual parameter is to reality 
[Gro99, RGHOO]. 
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The SISO further defines precision as a 1. The quality or state of being clearly depicted, definite, measured or calculated, 2. A qual¬ 
ity associated with the spread of data obtained in repetitions of an experiment as measured by variance, the higher the precision, 3) A 
measure of how meticulously or rigorously computational processes are described or performed by a model or M&S (SISO) [RPGOO]. 

Precision is the degree of exactness or discrimination with which quantity is stated (e.g., precision of 3 decimal places versus a 6 
decimal palace precision) [IEE90]. 

Timeliness is impacted by the maximum magnitude of error between the parameter with and without the missed updates, caused by 
communications delays or an excessive computation time, supports quantifying the impact of timeliness on a specific parameter [Gro99, 
RGHOO]. 

Time in a simulation may be managed with various techniques such as continuous time, time step, discrete event, etc [Gro99, 
RGHOO, RGHOO]. 

Error is the difference between a calculated, measured, or observed value and a correct value (SISO) [RPGOO]. 
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software) becomes apparent, adding that if the error causes the system to perform a func¬ 
tion incorrectly, a failure occurs. According to [Gro99, RGHOO], any review of simulation 
fidelity involves factoring in error. Possible M&S error sources include imperfect meas¬ 
urement of input data, deviation from correct data, imperfect algorithms, and the finite 
limitations of logical and computational processes. 

A report on software error analysis [PW93] sponsored by the NIST found a 
wide variation of error techniques for high integrity software and proposed the develop¬ 
ment of an organizational error analysis database identifying the most effective techniques 
to support error detection. Moreover, the impact of error in software-intensive systems, 
including simulation software is staggering. Of the $59.5 billion lost annually to the U.S. 
economy do to prevalent software errors; half of the cost is borne by the develop¬ 
ers/vendors, with the other half of the costs paid by the users [RTI02]. Over half of these 
errors were uncovered late in the development process or while in use after the sale 
[RTI02]. 


f. Resolution 

A significant M&S community issue includes the development of commu¬ 
nity standards defining resolution of model representations for use throughout the life cycle 
of the systems. [DoD98] defines resolution^^^, while [RGHOO] view resolution as “the ex¬ 
tent to which the M&S models each aspect of the real world”. A definition in the web- 
based [RPGOO] expanded the meaning of resolution . The [RPGOO] definition also ad¬ 
dressed granularity or the reduction of something into related parts or components. 

[HNP97] propose that the question of detail affects simulation resolution, with both factors 
related to fidelity. [Mey98] finds resolution erroneously used to describe the result of some 
measurement, rather than correctly identifying the resolution of the device taking the meas¬ 
urement (e.g., a sensor). 

[Mey98] also proposes that M&S resolution like fidelity, derives from the 
context of a simulation, and is a measure of the minimum degree that the constituent mod- 
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Resolution is the degree of detail and precision used in the real world aspects in a model or simulation [DoD98]. 
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The [RPGOO] defines resolution as 1) the degree of detail used to represent aspects of the real world, or a specified standard or refer¬ 
ent by a model or simulation, 2) separation or reduction of something into its constituent parts; granularity (SISO) [RPGOO]. 
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els’ accuracy and detail correspond. The semantic relationship between the terms pre¬ 
sented in Figure 6-2 delineates the delta between the fidelity required by the application or 
tolerances that define the range of values for dependent and independent variables, and 
the fidelity present in a model in a simulation, the knowable quantity. Figure 6-2 also illus¬ 
trates that physical reality, either material or imagined, provides the basis for obtaining all 
knowledge of reality. 

Known reality manifests this body of knowledge. Known reality also pro¬ 
vides the source for referents supporting application requirements and model or simulation 
fidelity, and for abstractions of reality, in models and simulations. [HNP97] suggest that 
model resolution, a critical component when determining the usefulness of a simulation is 
conceptually closely related to fidelity, however, caution that one does not imply the other, 
and higher resolution does not necessarily increase fidelity. However, since resolution af¬ 
fects federation credibility supporting confidence in the results, the [lEEOOc] definition for 
resolution is the most appropriate. 

g. Detail 

Simulation detail, like resolution, also includes many facets and views. 
[RR83] state that the model level of detail describes the number of functions, or level of 
structure included in the model, while the extent, a similar concept, relates to the range of 
system functions within the simulation. However, [Mey98] defines detail as a measure of 
model complexity and completeness when compared to the characteristics of the physical 
entity represented by the model. 

[Fay98a, Fay98b] proposes a method for measuring the level of detail in a 
simulation based on identifying the important parameters and a sensitivity analysis ap¬ 
proach, which provides a categorized hierarchy of phenomena, model, and value parame¬ 
ters. [Had98] further suggests that the sophistication of the developers in the problem do- 


Tolerance; 1. The maximum permissible error or the difference between the maximum and minimum allowable values in the proper¬ 
ties of any component, device, model, simulation, or system relative to a standard or referent. Tolerance may be expressed as a percent 
of nominal value, plus and minus so many units of a measurement, or parts per million. 2. The character, state or quality of not interfer¬ 
ing with some thing or action [RPGOO]. 
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The IEEE HLA definition of resolution describes it as the smallest resolvable value separating attributes or parameter valuses that 
can be discriminated. Resolution may vary with magnitude for certain data types [lEEOOc]. 
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main factors significantly in the development of the appropriate level of detail in the appli- 
eation. 


h. Aggregation /Disaggregation 

Aggregating and disaggregating model representations present significant 
challenges to the development of eredible simulations [HNP97]. The many complex issues 
involving aggregation / disaggregation eomplicate the development of improved M&S 
eredibility. The [DoD02d] notes the lack of predominate type of M&S and identifies the 
need for varying levels of aggregation and flexible interoperability standards to meet di¬ 
verse eustomer needs. Seeking more definitive answers, [HJ95] explored several different 
approaehes to aggregation, and established requirements to develop consisteney between 
aggregated and higher fidelity representations, eoneluding that common intuition about 
cause, effeets and outcomes are often ineorreet. 

[BidOO] addressed additional areas for representing battlespace objeets as 
elements of other aggregate element representations, ineluding meehanisms for establishing 
aggregated representations on a permanent or temporary basis . Taking an object ori¬ 
ented approaeh, [HHPOO] proposed a proeess transitioning aggregated / deeomposed mod¬ 
els into an object-oriented representation by defining modules in an objeet oriented domain. 
The objeet oriented methodology supports other possible struetures proposed by [Tra93, 
CRB01,CM02]. 

The Department also requires credible, authoritative representations inelud- 
ing aggregated and disaggregated system representations of U.S., allied, friendly, paramili¬ 
tary, coalition, neutral, threat, systems, and for combat, combat support and combat service 
support systems and proeesses. This ineludes Department objeetives for establishing stan¬ 
dard taxonomies and common object classes for systems representations by FY2004 
[DoDOla]. Two basie approaches have evolved for dealing with resolution and fidelity is¬ 
sues involved in aggregation / disaggregation: the top-down and the bottom-up approaehes. 
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Aggregation is defined as the ability to group entities while preserving the effects of entity behavior and interaction while grouped 

[DoD98]. 
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Disaggregation refers to the ability to represent the behavior of the aggregated unit in terms of its component entities [DoD98]. 
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This scheme deals with situations where an object needs to be represented both individually and as part of an aggregation at the same 
time. [BidOO] 
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Other resolution issues involve the development of aceeptable algorithms for aggregating 
representations of a single system into groups of entities and disaggregation of grouped 
representations [You02a, You02b]. 

5, Model and Simulation Quality Approaches 

Department- and National-level decision-maker and stakeholder knowledge and 
perceptions of M&S quality directly impacts credibility in the simulation or federation, and 
confidence in the results. [Wal94] described five approaches for viewing and defining 
quality based on the perspectives of different stakeholders — transcendental, user-related, 
buyer-related, production-related, and process-related. Today, the Department employs the 
“transcendental” approach to M&S software development, attempting to maintain high 
quality without precisely defining or measuring the product. In lieu of precise measure¬ 
ments in M&S software, the M&S community labels model representations with subjec¬ 
tive, qualitative identifiers such as high, medium, or low fidelity. 

The user-related approach for user satisfaction described by [Wal94] for the De¬ 
partment’s software intensive, simulation development best fits the current verification, 
validation, and accreditation processes. The Department’s acquisition process most appro¬ 
priately instantiates the buyer-related viewpoints in the cost, schedule, and performance 
parameters of the acquisition process. Metrics, formal methods, and statistics represent the 
Department’s production-related approach, seeking to attain high quality by defining spe¬ 
cific characteristics of a product with precise measurements, while the many process mod¬ 
els (e.g. CMM, CMMI) provide the Department with process-related approach to prevents 
faults and minimizes rework. 

The production-related approach has the potential to improve confidence in large 
scale Department M&S software systems. [Boo02] identifies the “use of certain reasonable 
and quantifiable measures of product and process” [Boo02] to address the maturity of soft¬ 
ware engineering team that are indicative of the product quality. This dissertation identi¬ 
fies a new architecture-centered, product line-based approach to replace the existing tran¬ 
scendental approach to the Department’s M&S software development and acquisition, and 
addresses the processes for a successful implementation. 
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D. SUMMARY 

Chapter VI addressed software and simulation quality attributes affeeting heteroge¬ 
neous system representation anomalies and eredibility in Department M&S, especially in 
federation interoperability. The chapter reviewed the internal and external attributes of the 
ISO 9126-1 Quality model [ISOOl], testing for quality attributes, and M&S quality attrib¬ 
utes addressed in much of the current literature, including an overview of aggregation and 
disaggregation. The chapter included a discussion on M&S quality approaches. 

Quality is a major component of M&S credibility. The ISO Quality Model ad¬ 
dresses the six major internal / external quality characteristics: functionality, reliability, us¬ 
ability, efficiency, maintainability, and portability; and the four qualities in use characteris¬ 
tics, allowing users to achieve specified M&S objectives. The term “software quality” re¬ 
mains widely used today, and open to subjectivity, different views and various interpreta¬ 
tions of the definitions. Software quality continues to lag well behind the computer indus¬ 
try’s hardware engineering segment progress on quality, cost, and performance. While 
computer hardware engineering is well into the fourth computer era identified by [Pre97] 
and employs the production approach to quality according to criteria established by 
[Wal94], simulation software quality is best labeled as transcendental. The chapter also 
reviewed model and simulation quality attributes including fidelity, accuracy, resolution, 
error and detail, within a fidelity context [RPGOO]. 

This research found little evidence that the M&S software development community 
will achieve any consistent future success: 1) developing quality metrics for M&S quality 
attributes, and 2) applying them early enough in the M&S software analysis and design 
phases to quantifiably improve M&S software quality employing current paradigms. In 
addition, many of the past recommendations to improve the Department’s M&S portfolio 
listed below have been reiterated and reissued on repeated occasions, although it is uncer¬ 
tain if more time and more resources will improve the results: 

• Better verification and validation (V&V) techniques and accreditation processes, 

• Better discipline in the software development process, 

• Improved use of data, 

• Improved quality, 

• Improved documentation. 
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VII. SOFTWARE ENGINEERING EFFORTS SUPPORTING CREDIBILITY 


A, INTRODUCTION 

Chapter VII reviews three selected areas influencing credibility of large-scale, leg¬ 
acy simulations in the Department including software process improvement (SPI), the 
status of new major simulation software engineering initiatives (e.g., JWARS, JSIMS, 
JMASS) and the growing software engineering body of knowledge. The chapter also in¬ 
cludes a discussion of the challenges generated by Department contract management over¬ 
sight for achieving quality software and reducing heterogeneous system representation 
anomalies, Department institutional factors affecting M&S credibility, and recent Depart¬ 
ment software engineering education initiatives. In addition, research reveals a growing 
awareness that the Department needs qualified software engineers who understand the 
foundations of the software-intensive systems, including credible M&S, upon which we 
basing our future security, economic prosperity, and military preparedness. 

B, PROCESS IMPROVEMENT SUPPORTING M&S DEVELOPMENT 

Product-based activities evolved with the early software development industry, 
supported by artisans practicing software development as a craft, a dependence on heroes, 
and a largely anecdotal body of knowledge. Since the mid-1970s a view of building soft¬ 
ware as a process-based activity grew and began to challenge the traditional model of 
building software as product-based activity. This process-based activity concept drew 
heavily from the manufacturing sector with a theory that how the developer built the soft¬ 
ware affected the quality of the end product. Process-based activities now include a rapidly 
expanding field including process models, quality standards, and contractor evaluation cri¬ 
teria, including international process improvement, quality standards, and Department- 
based standards. 

According to [SG99] American corporate project managers for information tech¬ 
nology realized by 1996 the true cost, scope, and size of software project failures and fi¬ 
nally acknowledged that no technological or silver bullet solution existed. The problem 
and solution identified by [SG99] included people and processes. [Bro96] cited the need 
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for fundamental process improvement in industry/govemment software development or¬ 
ganizations driven by systemic failures to predict and meet cost and schedule targets, a 
need to reduce life cycle software costs, and an improved ability to identify and address 
technical risks. 

Process-based activities including process improvement is multifaceted and incor¬ 
porates continuous improvement process (e.g.. Kaizen) championed by Deming [Sch91], 
business process reengineering concepts [HC93], statistical process control for software 
improvement [FC99, WelOl], and the process enterprise [HamOlb]. [BE99] addressed 
methods for improving the process and product by optimizing the software product 
throughout the life cycle, identifying follow-on efforts including product lines for further 
exploration. Process-based activities also entail event-driven learning approaches for train¬ 
ing [HilOl], and project management processes [Fra94], which provide additional perspec¬ 
tives on the various dimensions of effective processes. [She97] noted the large set of proc¬ 
ess frameworks caused confusion and identified six specific compliance framework catego¬ 
ries: 

• Standards and Guidelines, 

• Process Improvement Models and Internal Appraisal Methods,’ 

• Contractor Selection Vehicles, 

• Quality Awards, 

• Software Engineering Eife-Cycle Models, 

• System Engineering Models [She97]. 

Standards and guidelines included U.S. military standards (e.g. MIE-STD 498), 
commercial standards (e.g.. Electronic Industries Association (EIA) IS 632), and Interna¬ 
tional Standards Organization (ISO) standards including the ISO 9000 series for quality 
systems and ISO Software Process Improvement Capability determination (SPICE), an in¬ 
ternational standard for software process assessment. Process-based activities generally 
defined the characteristics of good processes, and did not prescribe the how to implement 
the processes. The 1991 introduction of the software Capability Maturity Model (CMM) 
for software (SW-CMM) [PWC+95] built on product quality principles promulgated since 
the industrial age production line practices of the 1930s. The objective of the CMM is to 

achieve defined, managed, and optimized processes, with the expectation that improved 
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process maturity [MCR+00, BBF+01] will also yield a higher quality software product 
[BE99]. 

Follow-on process improvement/maturity products including [Hum98a, Hum98b, 
Hum98c] based on the same underlying CMM appeared for other closely related disci¬ 
plines including Software Aequisition [IEE98c] CMM [FCF-l-96, FDR-l-98, Gal99, SPM99, 
GFOO], Systems Engineering [H097, Hum98a, Hum98b, Hum98e, Hum99, MCOla], Inte¬ 
grated Product and Process Development [Zit97], People CMM [HB98e], the Personal 
(PSP) / Team Software Proeess (TSP) [H097, Hum99, VFS+99, MCOla] and most re¬ 
cently, the Capability Maturity Model Integration (CMMI). In a similar initiative, 
[MCR+00] developed a five-stage maturity proeess similar to the CMM for U.S. Govern¬ 
ment investment management of information technology. [WPL02] addressed the possibil¬ 
ity of improving M&S credibility, eomparing and contrasting CMM key praetiee areas with 
several common V&V methods, while [Rie02] discussed the possibility of the a M&S 
CMMI to better identify mature companies to develop and maintain M&S produets. 

The Capability Maturity Model for Software (SW-CMM) [PWC+95] supports or¬ 
ganizational development of improved, common, software proeesses based on a framework 
of best praetices, which impact productivity, performance, cost and customer satisfaction. 
There are five CMM maturity levels 1-Initial, 2-Repeatable, 3-Defined, 4-Managed, and 5- 
Optimizing, supported by specifie key processes areas (KPA) established at eaeh maturity 
level. For instance, KPAs for level 2-Repeatable inelude requirements management, soft¬ 
ware project planning, software project tracking and oversight, software subeontract man¬ 
agement, software configuration management, and software quality assurance. 

1 ” 7 ^ 

Acquisition agencies may specify a SEI Software Capability Evaluation (SCE) 
method or Software Development Capability Evaluation (SDCE) method employed by the 
Air Foree, to have a third-party examiner review a competitors strengths and weaknesses in 
order to reduce risk to the aequiring agency. TG99] compared and contrasted both evalua¬ 
tion teehniques noting both methods were time and resource-intensive for both the contrac¬ 
tor and the government. In addition, [OSOO] identified challenges and inconsistencies in 
the SCE proeess and suggested improvements to restore validity to the proeess. Confi- 

Software capability evaluations (SCE) are formal, systemic methods for assessing a contractor’s software development process 
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dence in the evaluation proeess is critieal since the most critical Department acquisition 
projects, termed ACAT 1 programs, must undergo an SCE or develop a risk mitigation plan 
[Gan99]. 

The debate over the value of software process improvement continues today. Al¬ 
though [FalOl] cites a draft document defining the requirements for process evaluation 
methods to improve process definition, [Mos02] contends that quality of software process 
improvement varies significantly within the Department today, and exhibits less disciplined 
processes when compared with the early 1990s. Concerned with the Department’s soft¬ 
ware acquisition practices, [AS03] implements Section 804 of the Bob Stump National De¬ 
fense Authorization Act for Fiscal Year 2003 requiring the establishment of software ac¬ 
quisition process improvement programs by Department agencies with a substantial soft¬ 
ware component, supporting a Department objective to improve the acquisition of soft¬ 
ware-intensive systems. 

In an effort to improve quality, quality award initiatives evolved based on the prem¬ 
ise that the quality of the processes used to develop and maintain a software system signifi¬ 
cantly influenced overall quality. Several organizations developed award programs to im¬ 
prove the quality situation, including the Malcolm Baldridge National Quality Award, es¬ 
tablished by the U.S. Government in 1987, and the European Quality Award [She97]. The 
apparent proliferation of these models had several consequences [Cla97], which resulted in 
the initiation of the CMM Integration (CMMI) project with staged representations 
[FAB+99a, FAB+99b], and continuous representations [FAB+99c, FAB+99d, and Ric02]. 
The development of the Capability Maturity Model-Integrated-Systems/Software Engineer¬ 
ing (CMMI-SE/SW) built on three initial disciplines: software engineering, systems engi¬ 
neering, and integrated product and process development. 

Organizations with software process improvement programs have reported benefits 
in product quality and productivity, corroborating increased product quality and decreased 
product costs results experienced in other sectors [BBF+01]. [Whi99] addressed the impor¬ 
tance of VV&A for quality software engineering, and illustrated where W&A processes 
integrated into the development process improved cost, performance and schedule. [Bal98, 
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PW93] also noted the importance of establishing an M&S software quality assurance pro¬ 
gram, such as the program detailed by [STSOO]. 

Currently, the software acquisition management (SAM) model remains a separate 
model awaiting integration into the current CMMI framework. [San99] identified im¬ 
proved process and product discipline supporting improved software engineering practices 
[DL93, Pfl98]. Advocating continued professional development, [Whi99] endorses con¬ 
tinuing training opportunities, and provides an outline for training M&S practitioners. 
Achieving these objectives requires a disciplined software engineering approach, mature 
software acquisition management processes [IEE98c, SPM99, SSJ+00], the capability to 
evolve and improve the current Department portfolio of legacy M&S, and maybe most im¬ 
portantly, management commitment supporting transformation and reengineering initia¬ 
tives. 

Process maturity closely correlates to improved system and software engineering 
practices and supports the management of large software projects with defined consistency, 
and process repeatability. [McGOl] discusses the primary benefits and business case ad¬ 
vantages for SPI from a properly run SPI program including improved cycle time, reduced 
software development costs, improved product quality, reduced maintenance costs, im¬ 
proved worker morale, and increased business competitiveness. [McGOl] also identified 
possible impacts on secondary business case metrics from improved SPI practices includ¬ 
ing projected sales, average historical penalties (e.g., contract work), personnel turnover 
costs, employee development costs, repeat business, and a final category, the risk likeli¬ 
hood with and without software process improvement. 

Although process improvement results are not yet conclusive, there are positive in¬ 
dications [McGOl]. In dealing with projecting transitions to various maturity levels a lin¬ 
ear correlation was established between the Quantitative Software Management^^^ (QSM) 
quantitative Productivity Index and the SEI CMM [Put92]. The [Put92] analysis concluded 
that the number of companies in the two highest maturity levels will be quite small for the 
near term period, unless a dramatic change in the rate of improvement occurs. In a follow- 


177 

QSM has been in business capturing management numbers for software development activities since 1978 and has a very large 
database populated with the complete spectrum of performers from poor to very good. 
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on paper [Put93] developed the expeeted eeonomie benefits of moving up the SEI seale. In 
later researeh, [LFT95] studied the eorrelation between the CMM and software developer 
performanee, eiting improved eost and sehedule performanee with inereasing process ma¬ 
turity and validating a correlation between project success and CMM ratings, suggesting 
that CMM ratings may be indicative of future success. 

Continuing research by [Cla97] involving one hundred and twelve projects in the 
sample concluded in part that software process maturity was a significant factor affecting 
software development effort, and furthermore, that a one increment change in the process 
maturity rating, after normalizing, resulted in a “15 to 21% reduction in efforf’ [Cla97]. 
[Cla97] provides a significant assessment of the effects of process on software develop¬ 
ment efforts, and concluded the CMM is well defined and establishes criteria to evaluate 
processes, and proposes that process maturity should be a factor in all cost models. 

The SEI compiled a comparison of process maturity profiles by a subset of the soft¬ 
ware community from 1996 [SEM97] through 1999 [SEMOO] revealing: 

• The number of organizations initiating software process improvement continues to 
increase [SEM97, SEMOO], 

• The proportion of commercial and in-house organizations initiating software proc¬ 
ess improvement are increasing [SEM97, SEMOO], 

• Manufacturing industries are conducting the most software process assessments 
[SEM97, SEMOO], 

• The service industry now conducts almost as many software process assessments as 
the manufacturing industries [SEMOO], 

• Nearly half of the reporting agencies have less than 100 software personnel 
[SEM97, SEMOO], 

• The overall community profile continues to shift towards higher maturity [SEM97, 
SEMOO], 

• The trend towards higher maturity profile for offshore organizations compared to 
U.S. organizations continues [SEM97, SEMOO], 

• Software Quality Assurance is the least frequently satisfied key process area (KPA) 
among organizations assessed at level 1 [SEM97, SEMOO], 

• Integrated software management, organization process definition, and training pro¬ 
gram are the least frequently satisfied level 3 KPAs among organizations assessed 
at level 2 [SEM97, SEMOO], 

• Higher process maturity has been reached among those organizations reporting re¬ 
assessments [SEM97, SEMOO], 
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• All groupings exhibit a similar pattern for moving from maturity level 1 to level 2 
and from level 2 to level 3. Furthermore, level 2 to 3 tends to be faster and have 
less variance [SEM97], 

• For organizations that began their CMM-based SPI effort in 1992 or later, the me¬ 
dian time to move from: 

■ Maturity level 1 to level 2 is 25 months, 

■ Maturity level 2 to level 3 is 23 months, 

■ Maturity level 3 to level 4 is 36 months [SEMOO]. 

[PGWOO] notes the seventy-one level 4 or level 5 appraised organizations as of Eebruary 
15, 2000, continued to grow since 1992 when there were no level 4 appraisals and only the 
IBM Onboard Shuttle team rated level 5 using the SCE method. 

The Department’s draft M&S strategic plan [DoD03a] supports accelerated organ¬ 
izational, operational, business, and process reforms. Current challenges to software proc¬ 
ess improvement implementation include the costs to implement a SPI program, ensuring 
development team follows processes, establishing process metrics, and maintaining process 
consistency. Common organization barriers to a SPI program include: a lack of resources, 
inadequate sizing of the SPI effort, lack of senior management sponsorship, middle man¬ 
agement apathy, and staff tension [SEI95]. Building on the issues behind poor software 
quality identified by [Dem95a] and [You93], [EvaOO] suggest that it is not only an issue of 
improving the software processes, but also changing the organization or project cultures 
into an atmosphere supportive of process improvement. 

In the late 1990s, a countering effort, the Eightweight / Agile process movement 
(e.g.. Extreme programming (XP), Adaptive software development) evolved in a counter¬ 
ing effort to the plethora of process models which restricted creativity, according to the Ag¬ 
ile process school of thought. The XP School reduces software development processes to 
the bare minimum [UHC02]. 

C. CONTRACT MANAGEMENT 

Contract management oversight [MBG+02] is a critical process for an organization 
acquiring software. The challenges of managing the myriad of contracts supporting the 
Department’s IT initiatives, including M&S software support continue to grow. Eederal 
Government spending on IT services almost doubled from fiscal years 1997 to 2001, in- 
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creasing from $9 billion to $17 billion, with the Department of Defense remaining the larg¬ 
est single aequirer of IT services and increasing spending by about forty-one pereent 
throughout the period [KSZ03]. During the same period spending on IT services from the 
General Service Administration (GSA) greatly inereased with spending on the GSA federal 
supply service schedule growing from approximately $405 million to $4.3 billion [KSZ03]. 

[DoD03a] eontinues to support the objeetive of increased information sharing and 
eollaboration within the Government (e.g., federal, state, loeal), academia, and industry 
However, the inability of the government to establish M&S as contraet deliverables, the 
absence of M&S requirements in many eontract proposal plans, and the growing number of 
long-term eontraets reduee the potential gains [HBD+01]. Another area of concern in¬ 
volves the possible ineffectiveness of the SCE process [FS94] for eontractors identified by 
[Mos02]. In addition, [HBD+OI] addressed several other systemie government-contractor 
issues impeding progress, also noted by other authors as indieated: 

• Collaborative environments misunderstood, 

• Contractual obligation constraints or restrictions [KMOO], 

• Only fifty percent of the reviewed programs identified M&S in the prime contraetor’s 
statement of work, 

• Contraetor ownership of reviewed M&S was nearly seventy-five percent, 

• Issues with proprietary M&S [Sul02], and a eorollary ehallenge—M&S products (e.g., 
eoneeptual models) as a eontaet deliverables [Pae02d], 

• Lack of management visibility into programs, 

• The lack of work breakdown structures [HBD+01 ]. 

D, SOFTWARE ENGINEERING OF MODERN DOD M&S 

[Ham96, and HNP97] note that software engineering discipline overlaps and inter¬ 
twines with simulation implementation and the system modeling proeess, which supports 
the simulation design. While reduced costs and improved performanee have marked the 
evolution of hardware and network products in accordance with marketplaee competition 
and Moore’s Law [Moo65], a recent study eommissioned by the Department of Com- 
meree’s National Institute of Standards and Teehnology [RTI02], concluded that software 
errors are prevalent and eosts the U.S. economy an estimated $59.5 billion dollars annually, 
a detrimental figure whieh equates to about 0.6 percent of the annual gross domestic prod¬ 
uct. 

166 



This is a systemic Department issue. The Department created several organizations 
and task forces over the past twenty-five years to evaluate methods for improving software 
quality and reliability, identify better software management processes, establish sound 
software engineering practices, reduce software development cost, and improve perform¬ 
ance and schedule metrics [CHK90]: 

• In 1979 a Joint Logistic Commanders Joint Policy Group on Computer Resources 
Management, concerned with adequate software support after the development pe¬ 
riod, evaluated post-deployment software support (PDSS) procedures supporting 
the transition and operational phases of the software system life cycle. The Depart¬ 
ment historically experiences the highest life cycle costs during the PDSS of a soft¬ 
ware system, 

• The Very-High Speed Integrated Circuits (VHSIC) Program created by the De¬ 
partment in 1980 supported improved delivery time and performance of military in¬ 
tegrated circuits with the development and insertion of military-qualified VHSIC 
chips into weapon systems. This early approach to a military hardware problem 
employed many of the product line principals explained later in the research and 
applied to the software challenge to fully use the hardware capabilities, 

• The Software Technology for Adaptable Reliable Systems (STARS) established in 

1983 by the Department investigated methods to reduce software development 
costs, increase software system reliability, improve software automation techniques, 
and identify the advantages of software reuse, 

• The Department contracted with the SEI, located at Camegie-Mellon University, in 

1984 to investigate new software technology, analyze software development 
environments, and provide software and system engineering education [BBG+03], 

• Finally, several Defense Science Boards task forces and summer studies since 1980 
reviewed multiple defense programs and recommended changes to the Depart¬ 
ment’s attitudes, policies, and practices regarding software development and acqui¬ 
sition [CHK90]. 

Today, many of the Department’s software-intensive systems lack quality 
[HNC+00], while software costs identified by [Coh94, SG94, SG99, RJI02] continue to 
increase. The persistent systemic issues of late software deliveries; coupled with poor- 
quality continues to grow [Bow02, Mic02a, Mic02b]; and stymie the demand for reliable 
software systems. [KMOla] confirmed a causal link between simulation credibility and the 
quality of the software engineering processes and practices involved in simulation software 
development and maintenance practices identified in many of the earlier studies and re¬ 
ports. Achieving simulation credibility and confidence in M&S results also depends on the 
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quality and interoperability in the underlying software, hardware, networks and software 
development proeesses and produets. 

A preponderanee of researeh indieates software engineering maturity laeks disei- 
pline and lags signifieantly behind other engineering disciplines. Our research suggests 
that simulation software development may be more complex than the development of other 
types of software, since a model is an “abstraction^’^ or approximate representation of 
something” [GMS+96], where no model is ever completely representative [BCN96], and 
[Sha75] contends no model is absolutely correct. Disciplined development and the M&S 
verification and validation process is critical to the establishment of M&S software credi¬ 
bility and the establishment of confidence in the accreditation decision and the simulation 
results. To date, the Department has not achieved any long-lasting Department-wide quan¬ 
tifiable results indicating improved simulation credibility. 

Research suggests that quality simulation software, based on a disciplined software 
engineering foundation [Sha93, VFS+99] supports simulation credibility and the mainte¬ 
nance of user confidence in the results. However, documented systemic simulation soft¬ 
ware engineering'’^ issues [Arm64, PAD78, Che86a, HW97, GFOO, CraOla, RieOl] since 
the mid-1960s included the lack of adequate software requirements, software configuration 
management, verification and validation processes, software engineering processes, and 
software quality methods. Summarizing the simulation software portion of the Depart¬ 
ment’s software Tower of Babel, [BTE-l-93] stated. 

Software will continue to be a problem because it is here that all of system 
complexity hides. The software problem will continue to be the problem of 
exactly what does the system do [BTE+93]. 

Simulation software development also shares all the same cost, performance, 
schedule, and quality software development challenges [Gib94, Coh94, SG94, SG96c, 
SG99, NJLOO, BWOl] found in other domains, and adds the complexity of developing and 
validating conceptual models with credible model representations of reality to achieve con- 


Abstraction is the selective emphasis on detail: specific details are suppressed and those pertinent to the problem are emphasized. 
Abstraction mechanisms focus on high-level aspects of an entity while concealing details. Three commonly used abstraction mecha¬ 
nisms are classification, aggregation, and generalization [Sow88]. 
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Software engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation and mainte¬ 
nance of software; that is the application of engineering to software [BCC+Olb]. 
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fidence in M&S results. In response to these issues, the Department initiated three large, 
joint software-intensive simulation development projeets during the 1990’s: Joint Modeling 
and Simulation System (JMASS), Joint Simulation System (JSIMS), and Joint Warfare 
System (JWARS). The Department approved, funded and initiated the three systems to 
replaee Serviees and Ageneies legacy M&S, with planned objectives of meeting software 
development cost, performance, and schedule milestones, improving system interoperabil¬ 
ity, reducing life-cycle maintenance costs, and conforming to the new HLA standards. 

The Joint Modeling and Simulation System (JMASS) [Bro92, MeyOl] started in 
1990. JMASS is unlike other Department simulation software development programs, 
which plan to deliver a complete, integrated simulation. The JMASS program provides 
architectural components consisting mostly of the simulation engine and services, interface 
standards, and a set of GUI-based tools to build models to integrate the models with other 
architectural components and develop a simulation [MeyOl]. The major deliverables ex¬ 
pected from the JMASS effort include common digital simulation architecture; definition 
of standard interfaces; application of commercial standards; a CASE tool environment and 
M&S support for the entire acquisition lifecycle. 

The Joint Simulation System (JSIMS) [Mar97], conceived as the primary Depart¬ 
ment simulation tool supporting future joint-and Service-based training, education, and 
mission rehearsal roles and functions, began with an approved July1994 mission need 
statement. The JSIMS design envisioned a single, distributed, integrated simulation envi¬ 
ronment supported by a core infrastructure and mission space objects maintained in a 
common repository. However, by June 1999, [Ett99] declared that JSIMS had, 

“Serious funding, technical, and management problems...JSIMS remains a 
troubled program ”, with major architecture issues, and an inadequate 
management structure, requiring a major new baseline of the acquisition 
program, increased Department management oversight required, and the 
need for continued Department program support” [Ett99]. 

The Initial Operating Capability (IOC) for JSIMS, originally scheduled for Decem¬ 
ber 1999, experienced several delays with the last JSIMS IOC scheduled for March 2003. 

The JSIMS PEO declared that (1) the current program was unexcutable; (2) the acquisition program baseline was breached and must 
be rebaselined; and, (3) the IOC for JSIMS would slip at least 11 months [Ett99] 
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The life-cycle program funding profile projects $1.6 billion for the period of 1996 through 
2007 [UMS+Ola], The JSIMS future is unclear according to [UMS+Olb] citing a lack of 
assurance that the Department JSIMS Milestone Decision Authority would be able to make 
future informed investment decisions on JSIMS, based on reoccurring historical problems 
including multiple schedule delays, performance issues, and increased costs. 

The Department approved a third major simulation development effort, the Joint 
Warfare System (JWARS) [MetOO, ManOl], in May 1995 to develop a state-of-the-art, 
multi-sided simulation of joint, campaign level warfare representing joint functions, proc¬ 
esses, doctrine, and component warfare operations of future warfare to support analysis. 
The JWARS high-level design objectives included plans to represent future warfare capa¬ 
bilities supporting force structure decisions, course of action analysis for the Combatant 
Commanders, and the development of operational plans from a joint operation perspective. 

The JWARS system objectives also included the need to provide a better assess¬ 
ment of C4ISR contributions, weapon systems, joint requirements and the analysis of mo¬ 
bility, logistic and contingency operations impacts on warfighting capability, and support 
for the analysis of investment alternatives. When software development began in April 
1997, the JWARS program did not require a formal conceptual model product, and subse¬ 
quently the V&V agent created a virtual conceptual model from available development ar¬ 
tifacts [MetOO]. Initial V&V efforts to include algorithm validation were unsuccessful ac¬ 
cording to [MetOO]. The lack of a simulation conceptual model has been a recurring sys¬ 
temic issue with Department simulation software development, a major reason for the cur¬ 
rent lack of simulation and federation credibility and general skepticism of the Depart¬ 
ment’s simulation results. 

The three major Department simulation software engineering efforts: JMASS, 
JSIMS, and JWARS are all over cost, behind schedule, and have yet to demonstrate they 
can achieve performance objectives. Partly based on these experiences, the Department 
implemented a new approach for the Joint Distributed Engineering Plant (JDEP) initiative 
to improve System-of-System interoperability: large-scale reuse [DahOla, DahOlb, DTOla, 
DTOlb, Rad02]. [WolOl] established the Department policy for mission critical C2 inter¬ 
operability for the Joint Task Eorce and below, including the JDEP. Rather than build the 
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JDEP from scratch, the Department is making use of large-grain reuse [CWS02], pulling 
the neeessary assets from aeross the Department’s enterprise to support integration, inter¬ 
operability, and meet information assuranee objeetives. 

Software engineering and the V&V proeesses support improved eredibility and eon- 
fidenee. [KMOla] identified the link between simulation eredibility and software engineer¬ 
ing proeesses and good model management praetiees. The Department’s verifieation proe- 
ess also “evaluates the extent to whieh the model or simulation has been developed using 
sound and established software engineering teehniques” [DoD98, DoD95, AR97]. Stan¬ 
dard software development proeesses and software engineering methodologies have been 
used to develop Department simulation software, ineluding the ineremental, prototype, evo¬ 
lutionary, spiral [Boe93, BoeOO], and re-engineering methods [RPGOO]. However, De¬ 
partment software development proeesses for the most part, laek the diseipline of the more 
mature physieal engineering fields [DoDOla], and also laek eonsistent enforeement meeha- 
nisms [HNC+00]. This statement may be signifieant sinee troubled or failed software pro- 
jeets, industry-wide, are still the norm, and the eosts for failure are staggering. 

Software, as a produet and the teehnology for delivering a produet, has made major 
eontributions to the twenty-first eentury world, but major systemie ehallenges eontinue to 
persist aeeording to [Rei93a, Rei93b], and many of these problems may eause unintended 
eonsequenees. While there has been marked progress in many areas. 

Many prior defieieneies eontinue to shape DoD M&S beeause they require 
long-term solutions and eompeting priorities that have slowed progress 
[DoDOla], 

[Kil02] eites the laek of eooperation from M&S developers as the “single greatest 
obstaele to developing a meaningful assessment of the eredibility and fitness of simula¬ 
tions” [Kil02]. As a result, [PaeOlb] eonsiders the state of eurrent praetiees for simulation 
software development, assoeiated M&S proeesses, and produet management inadequate. 
Although [Pre97] suggests we have a ehronie afflietion as opposed to a software erisis , 


Fitness: Providing the capabilities needed or being suitable for some purpose, function, situation or application [RPGOO]. 

The term “software crisis” was identified by the DoD in the mid-1970s [SCC+98] and has been associated with the complex prob¬ 
lems associated with producing software that was developed on time, within budget, operated properly, and was maintainable over its life 
cycle. More recently its was identified by [Gib94] as the need for a mature software engineering discipline supporting an information- 
age society, that he believes remains years, maybe decades away from achieving, even after fifty years of evolution [Gib94]. 
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he cites an aging software plant and several unresolved systemic software-related problems 
including: 

• Hardware advances continue to outpace software capabilities, 

• Software has not kept up with the demand for new or replacement programs, 

• Society is now dependent on the reliable operation of software as witnessed by the 
Year 2000 problem, 

• High software reliability and quality have not been achieved, 

• Poor design and inadequate resources [Pre97]. 

The software industry is also growing ever more important to the twenty-first cen¬ 
tury society. The demand for new software has grown so rapidly that the demand far ex¬ 
ceeds the supply and the following conditions prevail according to [JBC+99]: 

• Software is fragile, unreliable, difficult and labor-intensive to develop, test and 
evolve, 

• The Nation’s ability to produce software does not meet the demand, 

• The Nation depends on fragile, inadequate, software products developed with im¬ 
mature processes, 

• Current technologies to support reliable and secure software are inadequate, 

• Software continues to grow more sophisticated and diverse, 

• Software is replacing manual processes in many activities previously exercised 
manually by individuals, 

• The Nation needs to invest more in software research [JBC+99]. 

Historically, the software engineering community viewed software engineering 
processes as discrete activities. However, while there are discrete activities including en¬ 
trance and exit criteria, design, coding, and testing, most of the development process pro¬ 
gresses in a continuous fashion [STS99]. The continuous software engineering activities 
[STS99] include configuration management, quality assurance, verification and validation, 
maintenance planning, risk analysis and management, and software engineering project 
management. 

Software requirements for models, simulations, and federations include three major 
categories: the problem domain, the user domain, and the situation domain [RPGOO]. The 
simulation software requirements process articulated by [BL91, Som95, SG96a, SG96b, 
BIL97, Pre97, BW97, CS99, HHPOO, KGOO, MWCOO, RPGOO, HLOl] at the most basic 
level are properties, exhibited by a new or adapted M&S system to solve a particular prob- 
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lem. They may have different properties sueh as proeess, or system constraint, and product 
parameters, which include functional (capability) and non-functional requirements such as 
constraints and quality requirements. For instance, [Som95] suggests a reliability require¬ 
ment places constraints on the system architecture. 

Emerging requirements may be dependent upon the system interoperation, as op¬ 
posed to a single feature or component, and critically dependent on system architecture de¬ 
cisions. Requirements elicitation is seldom easy and requires familiarity with many tech¬ 
niques and an understanding of the impact of social, political, or economic factors on the 
different stakeholders’ view of the requirements. Many times requirements understanding 
continue to evolve during the design and development phases. The requirements then un¬ 
dergo analysis to understand the system components and how they interact with other com¬ 
ponents, establish baselines and prioritize the requirements, and finally develop the esti¬ 
mated cost. The root cause in the Ariane 5 disastrous failure was related to a breakdown in 
the requirements process [STS99]. 

Furthermore, software engineering processes, development and maintenance prac¬ 
tices and procedures may provide significant insights into M&S credibility [KMOla]. It is 
also important to review M&S in the context of the software engineering environment. As 
the Department evolves from procedural-based software development techniques to an ob- 
ject-oriented approach, [Mey98] suggests that a model may be defined as a “digital or 
software representation of a physical entity”, in an object-oriented paradigm comprised of 
entities with associated attributes and behaviors, while a simulation may be viewed as the 
“software framework or architecture within which physical entities are animated” [Mey98]. 
At Figure 7-1 is one version the evolving software engineering body of knowledge (SWE- 
BOK) [Moo99, BCC+Olb] supporting improved simulation credibility. 


Object-oriented is a method of implementation in which programs are organized as cooperative collections of objects, each of which 
represents an instance of some class, and whose classes are all members of a hierarchy of classes united by inheritance relationships 
[PMR94]. Includes object-oriented analysis, design and programming. 
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SOFTWARE ENGINEERING BODY OF KNOWLEDGE 


Software 

Software 

Software 

Requirements 

Design 

Construction 

Requirements 

Softw^e 

Reduction 

Engineoing 

Design 

In 

Process 

Concepts 

Complexity 

Requirements 

Key Design 

Anticipation 

Elicitation 

Issues 

OfEHversity 

Requirements 

Softw^e 

Structuring 

Anal>sis 

Stmcture& 

For 


Architecture 

Validation 

Requirements 

Specification 

Softw^e 

Use of 


Quality 

Extanal 

Requirements 

Anaiysis 

Standards 

Validation 

Softw^e 


Requirements 

Design 


Managemait 

Notations 



Softw^e 

Strategies 

&Ivfethods 



Software 

Software 

Software 

Testily 

Maintenance 

Configuration 

Management 

Concepts 

Ivfaintenance 


And 

Concepts 

Managanent 

Definitions 


OftheSCM 


N/hintenance 

Process 

Test 

Process 


Levels 


Software 


N/hintenance 

Configuration 

Test 

Techniojues 

Key Issues 

Identificaticm 


N/hintenance 

Software 

Test-Related 

Techniques 

Configuration 

Ivfeasures 


Control 

Managing the 


Software 

Test Process 


Configuration 
Status Acct 

Software 

Configuration 

Auditing 

Software 

Release and 
Delivery 


Software 

Ei^neerii^ 

Management 

Software 

Engineering 

Process 

Software 
Ei^neering 
Tod & Method 

Organizaticmal 

Managemait 

Software 

Eng Process 

Software 

Tools 

Process/Project 

Infiastructure 

Process 

Software 

Infiastructure 

Ivfethods 

Software 

Process 

Nfeasurement 


Engineering 

Measurement 



Process 

Definiticm 



Qualitative 

Process 

Analysis 



Process 



Inplementatitxi 



& Change 



Software 

Quality' 


Quality 

Concqjts 

Define & Plan 
For Quality 

Techniques 
Requiring two 
ormwe 

Support to 
Other 
Techniques 

Testing 
Special to 
SQAorV&V 

Defect Finding 
Techniques 

Measuranent in 
Software Quality 
Anal>sis 


Figure 7-1. The SWEBOK (From [BCC+Olb]) 

The software engineering process knowledge area is a dynamic, rapidly evolving 
field, with new paradigms and models constantly introduced. The Department’s approach 
to software evolved from a software developer to a software acquirer, and today the De¬ 
partment acquires most software as commercial-off-the-shelf products or contracts with the 
private sector for the development effort. However, [Mos02] cautions that government 
program managers developing the contract request for proposal and the government con¬ 
tractor world winning the contract awards do not understood commercial best practices. 

Conversely, [Mos02] contends the real commercial world contractors, bidding for 
private sector contracts know that best practices critical for success includes quality soft¬ 
ware, software architectures, and product lines, which provide a competitive advantage. 
[Mos02] concern echoes a recent Defense Science Board [HNC+00] report, which stated. 
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The major factor in successful software development is disciplined execu¬ 
tion. . .too often; programs lacked well thought-out disciplined program 
management and / or software development processes. [HNC+00]. 

At a macro level, [FerOl] notes that software product quality and progress in the 
Department’s weapon systems depend on a myriad of complex internal and external forces 
(e.g., Congress, DoD, Services), both within and outside the program manager’s purview 
[Bro74]. In an article by [JonOO] on the best and worst software development practices, he 
characterized the military software domain with good development methods, supported by 
good quality control, but executing “marginal or deficient project management method” 
[JonOO]. However, there are some positive indicators. [SG99] cites improved project man¬ 
agement, processes, and people as contributors to this positive trend, while interestingly 
enough, technology was not the problem, n or the silver bullet [Bro87, Bar02c] solution. 
[SG99] noted three factors contributing to this positive trend: smaller projects, better pro¬ 
ject management and greater use of standard infrastructures. 

Future technical, organizational, cultural, and managerial factors in the missile do¬ 
main simulation development domain cited in Table 7-1 will be significant challenges. In¬ 
stitutional change or transformation, led by early adapters requires reengineering of many 
institutions and processes, especially in the Department of Defense with its current and fu¬ 
ture strategy to defend the Nation built around information. Where once the prevailing 
school of thought in the Department’s information technology world centered on process¬ 
ing power and bandwidth issues, these problems have technical solutions and are now more 
a matter of implementation. Replacing these older problem sets are many new, complex 
issues including information operations, information assurance, and the security of personal 
information from misuse. 

Table 7-1 summarizes the Department institutional constraints adversely affecting 
M&S credibility today. Note the close correlation of the constraints cited in Table 7-1 to 
the major research areas identified in Figure 2-2. Significant persistent, organization, proc¬ 
ess, management, and technical constraints limit or adversely affect simulation software 
credibility in the Department [DoD95, San97b, Ett99, RPGOO, HNC+00, PacOlb, DoDOSa, 
Nor03]. Software engineering and software architecture skills are now emerging as critical 
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domain disciplines that must continue to mature the eorresponding bodies of knowledge, if 
the Department and the Nation are to exploit the full potential of computer teehnology, and 
set the foundation for the next great era in eomputing, the Produet Line Era. Produet Line 
Praetice [Nor03] areas address many of these eonstraints. 


Department Model & Simulation Credibility Constraints | 
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Table 7-1. Department Institutional Constraints Affecting M&S Credibility 

Future domain software engineers and domain software arehitects will need to un¬ 
derstand the viewpoints of multiple, diverse stakeholders at all levels of the Department 
Enterprise and synehronize the software arehitecture and software engineering solution that 
meets cost, performance, schedule, and other constraints cited in Table 7-1. Developing 
and maintaining curreney in the skill sets of domain software engineers and software arehi- 
teets grounded in the teehnieal foundations of the field (e.g., eomputer science, software 
engineering, software arehitecture, information teehnology), with experienee in multiple- 
domains and functional areas will also require eontinuous education opportunities and pro¬ 
gressive developmental assignments. 

E, SOFTWARE ENGINEERING EDUCATION INITIATIVES 

The Department’s draft strategic M&S plan [DoD03a] includes the a strategic goal 
for awareness, edueation, training, and collaboration, and proposes M&S education pro¬ 
grams through the postgraduate level, providing Department employees with hands-on ex¬ 
perience through on-the-job-training, internships, and rotational assignments. Systemie 
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information technology (IT) issues facing the Department today include the shortage of 
educated IT professionals [HP99a, PH99, BynOO, MatOO, RBB+00], with a special empha¬ 
sis in the area of information assurance [Ste02b]. 

Improving the Department’s IT competencies includes many disciplines and emerg¬ 
ing bodies of knowledge, including software engineering. A growing body of literature 
addresses the complexity of the software engineering field including: 

• Studies of the best software engineering training practices [MTC96, Wie96b], 

• Software engineering education philosophy and alternative approaches [Par99, 
Bec99, BerOOb, CDC+00, HisOO, RBB+00, ShaOOb, RieOI, CT02, BM02, HH02a, 
HH02b, SBM02, DT02, Jen02, BK02b], 

• Teaching software engineering processes [Wie96b, BKOIb, BCH+02, BKK02], 

• Collaborative and distributed software engineering education to transform surplus 
engineers from traditional areas into software engineers [BPD02, HH02b, HMR02, 
UHC02], 

• Applying the evolving software body of knowledge [Wie96b, TH02], and 

• Adding a software engineering capability to existing military system engineering 
disciplines [Flo02b]. 

Emerging from the intellectual stimulus and software engineering studies are direc¬ 
tories of industry and university software engineering collaboration programs [Bec99], 
guidelines for software engineering education [BHH+99], certification programs [OSA+OI, 
computing curricula guidelines including software engineering [CDC+00], and new gradu¬ 
ate- and postgraduate-level software engineering education opportunities [NPS02]. 

F, SUMMARY 

In Chapter VII we reviewed three selected areas influencing credibility of large- 
scale, legacy simulations in the Department including process improvement, the status of 
new major simulation software engineering initiatives (e.g., JWARS, JSIMS, JMASS) and 
the growing software engineering body of knowledge. The chapter also included a discus¬ 
sion of the challenges generated by Department contract management oversight for achiev¬ 
ing quality software and reducing heterogeneous system representation anomalies. Depart¬ 
ment institutional factors affecting M&S credibility, and recent Department software engi¬ 
neering education initiatives. In addition, research reveals a growing awareness that the 

Department needs qualified software engineers who understand the foundations of the 
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software-intensive systems upon whieh we basing our future security, economic prosperity, 
and military preparedness, including credible M&S. 

The first component supporting M&S credibility is process improvement. By 1996, 
American corporate information technology project management realized no technological 
or silver bullet solution existed to reverse the tide of software project failures. The true 
scope of the problem and solution included processes and people. In an effort to improve 
quality, process improvement models including the Software Engineering Institute Capabil¬ 
ity Maturity Model Integration (CMMI), Capability Maturity Model (CMM) for software. 
Software Acquisition CMM, Systems Engineering CMM, ISO 9000, and the draft ISO 
15504 standard for software process assessment (SPICE) initiatives evolved. These proc¬ 
ess models matured based on the premise that the quality of the processes used to develop 
and maintain the software significantly influenced the quality of a software system. Initial 
indications suggest that process improvement methods contribute to improve overall soft¬ 
ware product quality and productivity. However, the Department inconsistently supports 
software process improvement. 

Contract management oversight is a critical process for an organization acquiring 
software. The challenges of managing the myriad of contracts supporting the Department’s 
IT initiatives, including M&S software support, and improve collaboration, continue to 
grow. Eederal Government spending on IT services almost doubled from fiscal years 1997 
to 2001, increasing from $9 billion to $17 billion, with the Department of Defense remain¬ 
ing the largest single acquirer of IT services and increasing spending by about forty-one 
percent throughout the period. During the same period spending on IT services from the 
General Service Administration (GSA) greatly increased with spending on the GSA federal 
supply service schedule growing from approximately $405 million to $4.3 billion. 

Research indicates software engineering maturity lacks discipline and lags signifi¬ 
cantly behind other engineering disciplines. Our research also suggests that simulation 
software development may be more complex than the development of other types of soft¬ 
ware, since a model is an “abstraction or approximate representation of something” 
[GMS+96], where no model is ever completely representative [BCN96], and [Sha75] con¬ 
tends no model is absolutely correct. Disciplined development and the M&S verification 
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and validation process is critical to the establishment of M&S software credibility and the 
establishment of confidence in the accreditation decision and the M&S results. However, 
the Department has not achieved any long-lasting Department-wide results improving 
simulation credibility. Quality simulation software, based on a disciplined software engi¬ 
neering foundation [Sha93, VFS+99] supports simulation credibility and the maintenance 
of user confidence in the results. However, documented systemic simulation software en¬ 
gineering issues [Arm64, PAD78, Che86a, HW97, GFOO, CraOla, RieOl] since the mid- 
1960s included the lack of adequate software requirements, software, configuration man¬ 
agement, verification and validation, software engineering, processes, and software quality. 

The next component of this research supporting M&S credibility is a review of the 
case studies of three modern software engineering projects involving new Department 
M&S initiatives: JWARS, JSIMS, and JMASS. This review was significant for several 
reasons. First, these replacement systems are overdue. The new Joint M&S programs 
JWARS, JSIMS, and JMASS, scheduled to replace many legacy systems are also over cost, 
and have reduced the original number of requirements and requirements to meet a future 
operational status. It is still to be determined if they will be useful for the emerging and 
rapidly evolving requirements of the 21®* century. 

Improving the Department’s IT competencies includes many disciplines and emerg¬ 
ing bodies of knowledge, including software engineering. A growing body of literature 
addresses the complexity of the software engineering field. The Department’s draft strate¬ 
gic M&S plan [DoD03a] includes the a strategic goal for awareness, education, training, 
and collaboration, and proposes M&S education programs through the postgraduate level, 
providing Department employees with hands-on experience through on-the-job-training, 
internships, and rotational assignments. However, systemic information technology (IT) 
issues facing the Department today include the shortage of educated IT professionals 
[HP99a, PH99, BynOO, MatOO, RBB+00], with a special emphasis in the area of informa¬ 
tion assurance [Ste02b]. 

For over twenty years the Department hoped for a “silver bullet” solution to resolve 
the “software crisis”. However, as hope for a “silver bullet” technical breakthrough to im¬ 
prove the Department’s software, including simulation software, waned in the 1990s, it be- 
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came apparent that other non-teehnieal solutions were required. A second underlying cause 
for the eurrent situation is the apparent diehotomy presented by the similar M&S faetors 
cited in Table 7-1 eoneurrently identified as the problem and the solution. In 1996 a major 
Department M&S study [PHP-l-96] identified three major types of systemic Department 
M&S challenges, technieal, cultural, and managerial; while concurrently the Simulation- 
Based Aequisition (SBA) initiative identified three similar factors, process, culture, and 
environment as SBA enablers. More recently the SEI endorsed three related practice areas, 
technical management, organizational, and software engineering to support the successful 
implementation of software produet lines. 

These faetors are significant since this methodology represents an institutional re¬ 
alization that the resolution to the twenty year old “software erisis” is not a “silver bullet” 
solution. Instead, the problem and the solution are multi-dimensioned, eomplex issues that 
require changes in all sectors of the Department. This ehange or transformation, led by 
early adapters requires a reengineering of many institutions and proeesses, espeeially in the 
Department with its current and future strategy to defend the Nation built around informa¬ 
tion. Where onee the prevailing school of thought in the information teehnology world 
centered on proeessing power and bandwidth issues, these problems are teehnically solved 
and more a matter of implementation. 

This realization will take time for society and the Department to truly assimilate and 
adapt to during the ongoing transition phase into the Information Age. We must however, 
also be mindful of the many Industrial Age vignettes where nations, armies, navies, or air 
forees were slow to adapt and the severe consequenees of that that failure in terms of defeat 
or loss of life. 
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VIII. SOFTWARE ENGINEERING OPTIONS TO IMPROVE CREDIBILITY 


A. INTRODUCTION 

Chapter VIII discusses traditional product lines, and introduces software product 
lines and software architectures. The chapter also reviews component development sup¬ 
porting a product line methodology, product line practice areas, and an overview of evolv¬ 
ing software architecture theory potentially applicable to improved M&S credibility and 
reducing heterogeneous system representation anomalies. 

B. PRODUCT LINES 

Historically, Industrial Age companies found that using common assets to build re¬ 
lated systems yielded significant market opportunities and improvements in time to market, 
customer satisfaction, product quality, meaningful metrics, and supported a competitive 
advantage for additional market-share. Henry Ford adopted the same concept when he in¬ 
troduced the assembly line, or product line method to the automobile industry, establishing 
the private industry product line model [MNJ-l-02]. 

In possibly the largest historical application of the product line concept, the United 
States private industry in World War II manufactured the majority of the ships, planes, 
tanks, and other major end items for the Allied war effort in numbers that the Axis powers 
could never match, establishing the same competitive advantage for wartime victory. 
Product lines continued to evolve in many Fortune 500 companies including Ford, McDon¬ 
ald’s and Boeing, albeit differently. In the Boeing product line example, the parts list of 
two entirely different aircraft, the 757 and 767 models, overlapped by approximately sixty 
percent [Cle99]. 

C. SOFTWARE PRODUCT LINES 

As the Information Age evolved and the corpus of software core assets accumu¬ 
lated, [Par76] noted. 

We consider a set of programs to constitute a family whenever it is worth¬ 
while to from the set by first studying the common properties of the set and 
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then determining the eommon properties of the individual family members 
[Par76]. 

The software product line concept builds on the experiences of component reuse including 

subroutines in the 1960s, modules in the 1970s, objects in the 1980s and component-based 

systems in the 1990’s. The SEI developed a product line definition from the successful 

private industry model and early analysis by [Par76]. A product line involves: 

A set of products sharing a common, managed set of features that satisfy 
specific needs of a selected market segment or mission [BFG+00, Nor03]. 

Software product lines building on product commonality however are a relatively 
new concept. The SEI Product Eine Practices Initiative built on the concept of a product 
line and introduced the concept of software product lines as “a set of software-intensive 
systems sharing a common, managed set of features that satisfy the specific needs of a par¬ 
ticular market segment or mission and that are developed from a common set of core as- 
sets in a prescribed way” [CN02]. The establishment of the software product line field¬ 
ing approach requires core asset'^^ development, or domain engineeringand product 
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development, also called application engineering using these core assets [BC96, Jon99, 
Nor03]. 

As an organization’s potential strategic information resource, the evolution of a 
software product line requires an understanding of the organization’s strategic plans, goals, 
business objectives, culture, technical acumen, risk threshold, and life cycle management 
plans [ISOOO]. This analysis supports the multiple related software product line families 
and products, each with concurrent versions and releases. A product family is also a set 
of products built from a common set of core assets [Nor03, CN02]. Although a product 
line does not require a product family, and a product family does not necessarily constitute 
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Core assets are those assets that form the basis for the software product line. Core assets often include but are not limited to, the 
architecture, reusable software components, domain models, requirements statements, documentation and specifications, performance 
models, schedules, budgets, test plans, test cases, work plans, and process descriptions. The architecture is key among the collection of 
core assets [Nor03]. 

^ A core asset is a software artifact used in the production of more than one product in a software product line. A core asset may be 
architecture, a software component, a process model, a plan, a document, or any other result of building a system [Nor03]. 

The domain engineering process develops software assets for one or more domains [Nor03]. 

The application engineering process develops software products from partial solutions or knowledge embodied in software assets 
[Nor03]. 

A product family method may yield the greatest efficiencies for developing product lines, depending upon market targets or feature 
relationships, although it is not a requirement [Nor03]. 
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a product line, the SEI definition of a software product line aeeording to [Nor03, CN02] 
implies a produet line of software products developed as a product family in a prescribed 
way, sinee this method leverages and amortizes prior investment [BC96, BBW+00], and 
potentially produees greater efficiencies, based on eeonomies of scope and economies of 
scale^^^. 

Software produet lines experience and growth in the eommereial software sector 
and the Department provides a growing body of software produet line case studies in the 
following areas: produet line investment analysis [Wit96, BBW+00], product line experi¬ 
ence and practice [BC96, WL99, DonOO, CN02], produet line organization and manage¬ 
ment [BCC+97, BCC+98, BCN+98, Cle99, BCD+00, CGF+00, WapOO, VWOO, TCOOO, 
MNJ+02], product line methods [CJBOO, KLD02, KNOO], product line proeess [WooOOa, 
FHROO], product line components (reuse) [Cle97, JNR98, ABMOO, BCSOO, PPOO, GriOO, 
CN02], produet line arehiteetures [DSOO, ProOO, ShaOOa], produet line tools and techniques 
[CJBOO, AOV+00, YMK+00, SSP+00, STOO, HOF+00], produet line domain engineering 
[ADD+00, FKK+00, DagOO, HSVOO, TPOO, MD02], and product lines in the Department 
[Jon99, BFJ99, BFG+00, BOS02, CDS02, Cam02]. 

[BFG+00, DDWOl, CN02, KFD02] suggest employing a software product line at 
one or more of the following levels: system^^'^, subsystem, or component, depending on the 
application domain, degree of commonality, and feature variability. A software product 
line notes [Nor03] includes multiple, related products with product-specific cycles of re¬ 
leases and versions, which evolve in eonsonance with the product line as a whole. The up¬ 
front cost of developing the first software product line member(s) often requires a signifi- 
eant investment, supported by a business ease and market analysis. Fielding a software 
product line includes the development of core assets and products coordinated by manage¬ 
ment activities. Software produet line development includes mining existing products for 
generic assets or core assets for later development use in an iterative manner, based on cur¬ 
rent needed capabilities, antieipated future requirements, and likely future produet variants 
[BOSOO, CDK+01, CN02]. 
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The condition where fewer inputs such as effort and time are needed to produce greater quantities of a single output (e.g., economy of 

scale), or a greater variety of outputs (e.g., economy of scope) [Nor03]. 
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A product line system is a member of a software product line [Nor03]. 
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1. 


Potential Benefit of a Software Product Line 


A product line approach to developing and deploying software-intensive systems 
offers great promise according to [WL99, BFG+00, CN02] for delivering higher quality 
systems in a shorter time and at reduced cost. According to analysis of the industrial sector 
experience by [BFG+00, LKK+00, DagOO, HSVOO, and CN02] a product line approach to 
software-intensive systems can save money and result in a faster time to field quality sys¬ 
tems. [Jon99, CN02] cite a number of organizations gaining order-of-magnitude improve¬ 
ments in efficiency, productivity, and quality through a product line approach. [BC96, 
Jon99, WL99, BBW+00, CN02] also note that even more important than cost savings is the 
fact that product line practice enables an organization to get its product to the market more 
rapidly. As an example, [BC96] provides a case study of a successful product line imple¬ 
mentation by the CelsiusTech Systems AB of Sweden in the area of large, embedded, real¬ 
time shipboard command and control systems. 

The SEI identified organizations benefiting from software product line practice in 
workshops cited by [BCC+97, Cle97, BCC+98, BCN+98, CW98, BCD+00, CGF+00] and 
case studies [BC96] including: 

• The Swedish naval defense contractor, CelsiusTech experienced a favorable rever¬ 
sal in the hardware-to-software cost ratio from 35:65 to 60:20, now favoring soft¬ 
ware, 

• Hewlett-Packard has metrics reflecting two to seven times cycle time improve¬ 
ments, 

• Motorola experienced a four times cycle improvement with 80% reuse on a pager 
product line, 

• Cummings Engine recorded a dramatic reduction in one case for a system build and 
integration from about one year to three days, 

• Thompson-CSE with air traffic control systems, 

• Alltel supporting commercial bank systems, 

• Ericsson, Noki, Eucent, and AT&T in communication systems, 

• Boeing in air flight software, and 

• The National Reconnaissance Office command and control systems for satellites 
[BC96, BCC+97, Cle97, BCC+98, BCN+98, WE99, BCD+00, BEG+00, CGE+00, 
CN02]. 

An organization’s business analysis supports the decision to establish a product line 
approach with product line goals, objectives, strategies and the development of a Concept 
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of Operations (CONOPS). The product line CONOPS [AIA93, CFM+96, Coh99, CN02] 
establishes an approach for achieving the organization’s product line goals of doing a job 
better, faster, and cheaper by focusing on efforts that reduce the costs and risks, associated 
with system development. The CONOPS is the system-users operational view of the sys¬ 
tem under development, and [Coh99] notes the CONOPS identifies in-house product line 
responsibilities for the government and commissioning organizations, and establishes ac¬ 
quisition/supplier relationships, determines the ownership of product line assets and defines 
access policies. 

However, there are also significant implementation issues with product lines identi¬ 
fied by [WL99, and EbeOl]. As a result, the SEI explored the range of issues and practices 
necessary for the successful implementation of software product lines [KZ96, Cle97, 
CN02], and developed several products and including an investment analysis methodology 
for software product lines [Wit96]. The SEI also sponsored a series of Product Eine Prac¬ 
tice (PEP) Workshops [BCC+97, BCC+98, BCN+98, BCD+00, CGE+OO] to share industry 
practices in software product lines and to explore the technical and non-technical issues 
involved with software product line ventures, including software engineering, technical 
management, and enterprise management functions, responsibilities, and issues. 

2, Challenges for Software Product Lines 

a. Developing Software Product Lines in the Department 

Any software product line implementation strategy in the Department must 
take into account the Department is an acquirer of software systems and not the software 
developer, and factor in product line practice areas supporting the Department’s acquisition 
process. Initial efforts to employ product lines in the Department’s M&S domain have 
been limited. The Air Eorce employed a structural modeling^^^ [ASC94, CB96] initiative 
to support the development of aircrew trainer simulator*^^ software for the B-2 Weapons 


Structural modeling has been in use since the mid-1980s during which it was expanded from a method for constructing a software 
architecture into an architecture-based development method. It includes the general engineering principles and technologies including 
prescriptive architecture, incremental development, and prototyping [CB96]. 

192 

A simulator is a device, computer program, or system that performs simulation or for training, a device which duplicates the essential 
features of a task situation and provides for direct human operation [DoD98]. 
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System Trainer (1986), C-17 Airerew Training System (ATS) (1990), Speeial Operation 
Forees (SOF) ATS (1991), and the Simulator Eleetrie Combat Training (1992) as a method 
to improve changeability and resolve the limitations of the previously employed data- 
driven architecture [CB96] noted that the simulation software was only a part of the B2 
trainer supporting the physical cockpit, the cueing system, and the display software, and 
not a complete architectural solution, as presented by this dissertation. A major point ad¬ 
dressed by [BC96] is that non-technical issues including business, personnel, management 
practices, cultural and process issues are equally as important to the success of a product 
line initiative as the technical issues. 

Several Department research and acquisition organizations recently explored 
product lines applicability within the Department’s M&S domain. The Naval Postgraduate 
School (NPS) developed a body of knowledge germane to this research initiative with re¬ 
search on the Janus model [JBL97, SLB-l-98, SLB-l-99, BLS+99], but it was limited in 
scope to a single model. The U.S. Army will use product lines as a method to replace out¬ 
dated legacy instrumentation and simulation systems used in support of the Maneuver 
Combat Training Centers with the next generation Objective Instrumentation Systems 
[DDWOl, MD02]. However, the [DDWOl, MD02] initiative appears more focused on the 
hardware components of a product line, and it is much too soon too soon to determine if it 
will be successful. [Bru98] also evaluated the product line methodology to determine the 
feasibility of the product line approach for software development in a single model. 

The Department recognizes the potential benefits of product lines along with 
an acknowledgment of the significant challenges to adopting this approach [Wit96, JNR98, 
BFG+00, JonOO, BOS02, Cam02, CDS02]. Many of the challenges stem from the fact that 
the Department is in the business of acquiring systems rather than developing them. Al¬ 
though product line practice works in industry, many attempts to emulate this success 
within the Department encountered problems [BNS97, BFG+00]. Product lines face im- 
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Changeability is defined as the ease with which the software standard can be maintained throughout its life cycle, and extends the 

concepts of modifiability and maintainability to include changes in requirements and specifications [CN96]. 
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The data-driven software architecture, a de facto standard since the 1960s increased the complexities and risk during integration and 
maintenance phases of the life cycle, most significantly data coherence problems stemming from spreading out the state information and 
communication of the state information across subsystems [CB96]. 
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The data-driven architecture consists of the executive code/scheduler, system routines, scheduling table, and the data pool, which is 
the data area shared by the system routines for storage and communication of state information [CB96] 
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plementations in the Department faee teehnical ehallenges and significant non-technical 
barriers such as culture and acquisition-related issues. The SEI identified several key is¬ 
sues an organization should address before moving to a product line approach for acquiring 
or developing software: 

• What constitutes the product line? 

• How will it be introduced? 

• What key organizational elements will be involved in defining, developing, and 
fielding the product line? 

• What is the relationship between product line assets and systems within the product 
line? 

• How will the architecture be developed and maintained? 

• What are the sources of software components [BNS97, BFJ99, BFG+00]? 

[BFJ99] recommends three product line acquisition activities for acquiring a 
product line in the Department’s acquisition environment. These activities include: 1) ac¬ 
quire architecture and other elements of an asset base to enable a product line approach; 2) 
acquire software products utilizing the asset base; and 3) acquire the services to maintain 
the asset base and support the development and enhancement of derivative products 
[BFJ99]. [BFJ99] also suggests that product line start-ups may be more successful in a sys¬ 
tem’s operational support phase, vice system start up phase. [BFJ99] also believes the stra¬ 
tegically positioned system sustainers may be motivated to adopt a product line approach, 
since they may have responsibility for sustaining and enhancing similar operational sys¬ 
tems. 


b. Component Development for Software Product Lines 

Components^^^ play a major role in many software-intensive systems 
[FG96, HBF97, BBB+00, CN02, FFA02b, SAG02]. There are several motivations for a 
components-based software approach to software: providing a basis for commerce in reus¬ 
able software to meet demand, facilitating the development of flexible systems, and reduce 
the time to design, implement, and deploy systems. As an indication of the need for soft- 
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A software component is an opaque implementation of functionality, subject to third-party composition, conformant with a compo¬ 
nent model [BBB+00]. 
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1 0-7 

ware components at the United States Federal Government level, the Office of Manage¬ 
ment and Budget recently established the Federal Enterprise Architecture Program Man¬ 
agement Office to remedy the lack of a Federal Enterprise Architecture (EEA) [EEA02b, 
HayOSb], a major barrier to the success of the E-Government initiative . The EEA pro¬ 
gram includes a Component-Based Architecture (CBA) [EEA02b] supported by tools, 
technologies, and standards facilitating component reuse, distribution, and cross-agency 
collaboration. 

The CBA framework [EEA02b, Gar02] builds on a tier-based architecture, 
employing layers to support the creation of components facilitating reuse and interoperabil¬ 
ity. The integrated CBA model [EEA02b] includes five architecture layers: Information, 
Technical, Security, Application, and Business employing an “Import and Export” method¬ 
ology supported by Extensible Markup Language (XML)^^^ schemas to support inter¬ 
agency interoperability and data sharing. The CBA [EEA02b] involves relevant industry 
standards (e.g.. Hypertext Markup Language (HTML), XML, XML Web Services, Simple 
Mail Transfer Protocol (SMTP), and Simple Network Management Protocol (SNMP)) pro¬ 
viding a foundation for interoperability, growth, integration, and expansion. 

The Department supports the CBA initiative in many areas [Lat97], includ¬ 
ing M&S. In the Department’s M&S arena, the DMSO initiated the Composable Mission 
Space Environment (CMSE)^'^'^ project and sponsored a Composable^^* M&S Workshop 
[ES02, Gar02, Lor02, Mat02, Pet02, Ves02, ZHS02] in July 2002 to explore ways compo¬ 
nent-based technology may reduce the manpower, time, and resources currently needed to 
execute Department simulations in a transformation environment with joint, interoperable, 
re-useable models [CA02, PEN02, RPK-l-02, ZHS02]. The Department’s on-going trans¬ 
formation process drives the transition requirement from a threat-based force applying sys¬ 
tem-focused, platform-centric approaches to a capabilities-based force employing mission- 
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At the Government Enterprise level a component is an application, capability, or service that leverages technology to perform a spe¬ 
cific business function [FEA02b]. 
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The President approved twenty-four priority E-Gov initiatives to meet the business needs of the Federal Government [FEA02b]. 
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XML is a platform independent, universal language used to support the stmcturing and integration of documents and data on the web 
with a flexible set of standards for tagging and classifying information readable by humans and data exchange systems [FEA02b]. 

The CMSE is an interrelated collection of enabling M&S technologies, tools, and procedures. 
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Composability is the capability to select and assemble simulation components in various combinations into simulation systems to 
satisfy specific user requirements [Pet02]. 
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focused network-centric approaches in how the Department trains, procures, equips, oper¬ 
ates and fights. The Workshop identified several specific recommendations for compos- 
able M&S standard development: 

• Develop and interchange standard for data and models, 

• Develop a Composable M&S Ontology, 

• Conduct a business case analysis, 

• Conduct further research in many less-understood areas including the impact of cul¬ 
ture and organizations on adoption of a composable M&S approach, 

• Engage basic research in component technology needs, 

• Identify VV&A issues [ZHS02]. 

The SEI recently initiated a component-based software engineering (CBSE) 
study to determine if they could extract predicted properties of a CBSE system of compo¬ 
nents made from the components themselves [BBB+00]. During the process [BBB+00] 
adopted the following narrow vision of CBSE: 

Component-based software engineering is concerned with the rapid assem¬ 
bly of systems from components where components and frameworks have 
certified properties', and these certified properties provide the basis for pre¬ 
dicting the properties of systems built from components [BBB+00]. 

Component-based systems rely on defined standards and conventions, or 
component model, and a support infrastructure, or component framework [BBB+00]. A 
component model specifies the component standards and conventions (e.g., component typ- 
ing, interaction schemes, resource binding ) whereas a component framework provides 
the developer with an implementation of services supporting or enforcing component 
model standards and conventions [BBB+00]. Although there still remains a lack of con¬ 
sensus on the contents of a component model, [BBB+00] suggest: 

• A uniform composition constraining how components interact if and only if they 
share consistent assumptions about what each component provides and requires of 
another component. Although some assumptions are unique to the component, a 
standardization of assumptions (e.g., component location, control flow, data encod¬ 
ing, communication protocol) reduces chances for accidental mismatch, which ad¬ 
versely affect composition. 


A component type is defined by the interfaces it implements. An interaction scheme specifies how components are located, commu¬ 
nication protocols, and quality of service attributes including security and standard transaction assumptions. Resource binding identifies 
the process of composing components in terms of binding a component to one or more resources [BBB+00]. 
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• Appropriate quality attributes, which depend on the software architecture and archi¬ 
tectural style^'^^ supports the desired quality attributes for a system composed from 
third-party components and the quality of service^®"^. 

• Deployment of components, application development, and the emergence of a vi¬ 
able market in third-party components are critical to the success of a CBSE envi¬ 
ronment. Historical examples abound where technologies survived and economi¬ 
cally prospered in the face of arguably superior technology (e.g., Apple versus Mi¬ 
crosoft, the Beta tape format versus the VHS tape format). This precondition for 
component composition suggests that components successfully transition from the 
component developer to an application developer’s composition environment, and 
finally, operate in the customer’s environment [BBB+00]. 

The second component concept advanced by [BBB+00] is the component 
framework. In an analogous sense, a component framework is comparable to an operating 
system, in which components are to frameworks what processes are to operating systems, 
and the framework manages resources shared by the components, and provide the underly¬ 
ing services enabling communication among components [BBB+00]. 

In practice, component frameworks include the Enterprise JavaBeans™ 
specification of servers and containers, the WaterBeans framework for real-time visualiza¬ 
tion of high-performance data streams, and Microsoft’s VisualBasic framework for visual 
composition of components [BBB+00]. A major challenge facing the successful imple¬ 
mentation of component frameworks is the issue of standard versus custom component 
models and frameworks. These requirements create competition between the maximum 
flexibility school of thought supporting different architectures, different styles, and differ¬ 
ent allocation of quality attributes based on the application; and the business case school of 
thought supporting the establishment of a viable market in software components 
[BBB+00]. 

Component interfaces are the foundation for component substitutability and 
significantly more complex than traditional system interfaces. This generated the idea of 
an interface contract [BBB+00] where, 

• A contract binds two or more parties, 

• The parties negotiate details of the contract before they sign the contract. 


The architectural style includes standardization of the types of components used in a system and their patterns of interaction 

[BBB+00]. 
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The quality of service specifies the patterns of interaction (e.g., Type of transactions, encryption methods) [BBB+00]. 
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• The contract prescribes normative and measurable behavior by all parties, 

• A contract cannot be changed unless signatories approve all changes [BBB+00]. 

[LG96, OHR99, BCSOO, PPOO, GriOO, ABMOO, and AB02] explain the de¬ 
sign of generic systems based on the reuse of software designs and components, supporting 
designs of families of systems or product lines based on commonalities, including compo¬ 
nent metadata [OHR99], within the product lines. In object-oriented design, a similar con¬ 
cept for extensible software system provides for a framework and specific plug-ins. Design 
quality is critical and some quality attributes may or may not be discemable at run-time, 
while other attributes relate to the architectural qualities such as conceptual integrity, cor¬ 
rectness, completeness, and build ability. Four general principles identified by [BL91, 
Som95, BW97, Pre97] impact software component construction: 

• Reduction of complexity, 

• Anticipation of diversity, 

• Structuring for validation, 

• Use of external standards [BL91, Som95, BW97, Pre97]. 

Composition is a key process within the software component develop- 
ment life cycle. The software composition design activity includes analysis of the soft¬ 
ware requirements [HBL97, BocOO, FHROO] and results in the development of the software 
product [BL91, Som95, LG96, LZB+96, Lat97, Pre97, Jac98, BBB+00, RPGOO, CooOlb, 
NSOl, AKN02]. Key enabling design techniques and principles include abstraction, cou¬ 
pling, cohesion, decomposition, modularization, encapsulation, information hiding, separa¬ 
tion of interface and implementation, sufficiency, completeness, and primitiveness. 

[Wil96, VTOl] discuss a number of key issues for consideration during the design phase 
including quality and the decomposition, organization, and packaging of the software com¬ 
ponents. [BW97, CA02, PLN02, RPK02] review other aspects of the systems behavior, 
which horizontally crosscut the system functionality, including concurrency, event han¬ 
dling and control, distribution, exception handling, interactive systems, and data persis¬ 
tence issues. 


Composition is the term used in component-based development to explain how so develop or assemble systems [BBB+00] 
[IEE90] defines software design as both a process and a product. 
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Several eritieal aspeet of eomponent-based software still require attention 
aoeording to [BBB+00, VTOl, and Voa02], ineluding the need for systems that will pre- 
dietably exhibit the quality attributes required of them (e.g., predietive eomposition) and 
address, 

• Changeable environment and dependeneies, 

• Variable usage, 

• Large number of small parts to manage, 

• Component flux (e.g., version ereep), 

• Developer ineonsistencies [VTOl]. 

Key teehnieal areas of eomponent-based development addressed earlier by 
[LG96, HBL97] and reeently by [BBB+00] inelude eomponents, interfaees, eomponent 
models, eomponent framework [Lat97, WilOl], eomposition [LG96, WilOl, AKN02, 
Voa02, KH02, PLN02], eomponent-based product line engineering [ABB+02], the compo¬ 
nent certification process [WR94, POPOO, CSS+01, GMOl, GSOl, JDLOl, MasOl, MCOlb, 
PSR+01, SWOl], and component wrapping [PSR+01]. In essence at this point on the ma¬ 
turity curve, the current body of knowledge on components, component trust and certifica¬ 
tion, component technology, and software architecture, 

• Raises concerns about properties of assemblies of components, 

• Lacks information about component behavior, 

• Lacks an understanding of the functional and extra-functional properties of systems, 

• Lacks the ability to determine properties of “black-box” component assemblies 
[CSS+01]. 

Currently there is not an established basis for how well component models 
and frameworks contribute to achieving the desired quality [WR94, LG96, JDLOl, VTOl, 
KH02, AKN02, Voa02]; nor is there any basis for assessing the quality of software compo¬ 
nent interfaces, composition, and component certification. [WR94, BBB+00, KH02, and 
Wr02] address components and component certification issues and analyzed component 
complexity and interfaces. [CTW98] evaluated software component licensing issues, while 
[Voa02] reviews several significant issues with composition practices. Consensus on the 
many solutions required for contentious component / composition issues do not appear on 
the near-term horizon. 
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3. 


Product Line Practice Areas 


[CN02, Nor03] defines a software produet line praetiee area as a body of work or a 
eolleetion of aetivities, whieh an organization must suoeessfully master to earry out the es¬ 
sential work of a software produet line. Many, if not the majority of the praetiee areas de- 
seribe aetivities, whieh are eritieal to the sueeess of any software projeet, and provide start¬ 
ing points for organizations to initiate and master aetivities and measure progress in adopt¬ 
ing a produet line approaeh. The praetiee areas in the framework divide into three eatego- 
ries: 1) software engineering praetiees, 2) teehnieal management praetiees, and 3) organiza¬ 
tional management praetiees illustrated in Table 8-1. 

Implementing software produet lines offers potential gains, and entails signifieant 
risks, ineluding teehnieal, organizational, and management issues. [Nor03] explains that 
building and aequiring a software produet line requires diseiplined engineering supported 
by mature teehnieal and organization management proeesses of universal essential aetivi- 
ties and praetiees, whieh the SEI developed into an online framework , a web-based 
doeument deseribing the eompeteneies needed to develop and field a sueeessful a software 
product line or any software-intensive system. 

A Framework for Software Product Line Practice, (Version 3.0) [Nor03] is a web- 
based tool introduced and designed to support the software community in software product 
line endeavors. Each version represents an incremental attempt to capture information 
about successful software product line practices. This information builds on the work of 
the SEI Product Eine Practice Initiative, including studies of organizations, which have 
built product lines, from direct collaborations on software product lines with customer or¬ 
ganizations, and from leading practitioners in software product lines. The SEI website de¬ 
fines all of the practice based upon ongoing software product line collaborations and the 
feedback from the community, future versions will build upon the current foundation and 
the growing body of knowledge, refining current knowledge, and describing a small num- 
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Framework in this context suggests a conceptual index, a frame of reference for the information essential for success with product 
lines [Nor03]. 
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ber of product line scenarios involving the development, aequisition, and/or evolution of a 
software product line. The Product Line Practiee envisions: 

• Product line development as low risk / high return 

• Teehniques for exploiting system commonalities and controlling variability with 
standard Department, government, and industry software engineering practices 
[Nor03]. 

There are several overlapping terms from previous information teehnology areas 
synonymous with produet line terms. [ADD+00, TPOO, and SehOOa] address core asset de¬ 
velopment activities, (e.g., domain engineering), while [BFG+00] addresses product devel¬ 
opment from eore assets, often eited as application engineering. The arehiteeture and 
eomponents are central to the set of core assets (e.g., platform), used to construct and 
evolve the produets in the produet line. Development is a generie term to describe 
aequiririgisajftarareeveral viable options for aequiring software [CN02, Nor03, MNJ02]. An 
organization may build the software in-house from scratch, mine existing legaey software 
assets (e.g., eore assets), purehase eommereial-off-the-shelf-software (COTS), or eommis- 
sion the development, normally through a contraet with someone to build the software 
[Nor03, MNJ02], or even eombine several options. The objeetive of the core asset aetivity 
is to support development of a production capability for products and requires three com¬ 
ponent activities: 

• Defining the product line scope, 

• Developing the produetion plan 

• Producing the core assets, [Nor03]. 

Table 8-1 identifies practice areas as a body of work or a collection of activities, 
whieh an organization must suoeessfully master to carry out the essential work of a soft¬ 
ware product line. The following seetions deseribe the three categories and the assoeiated 
praetiee areas. The number of products developed from core assets will form the founda¬ 
tion for metrics indicating the real potential for mining eore assets [BFG+00, BOSOO]. A 
short summary of each product line practice area follows. 
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PRODUCT LINE PRACTICE AREAS v4 i 

Software Engineering 

Technical Management 

Organization Management 

•Architecture Definition 

•Configuration Management 

•Business Case 

•Architecture Evaluation 

•Process Definition 

•Customer Interface Mgt 

•Component Devlopment 

•Product Line Scoping 

•Acquisition Strategy 

•COTS Utilization 

•Technical Planning 

•Funding 

•Mining Existing Assets 

•Technical Risk 

•Launching/Institutionalize 

•Requirements Engineering 

Management 

•Market Analysis 

•Software System 

•Tool Support 

•Operations 

Integration 

•Data Collection, Metrics, 

•Org Planning 

•Testing 

Track 

•Org Risk Mgt 

•Understanding Relevant 

•Make/Buy/Mine/ 

•Structure Org 

Domains 

Commission 

•Org Technology Forecast 

•Training 


Table 8-1. Product Line Practice Areas (From [Nor03]) 
a. Software Engineering Practice A rea 

Based on research to date, the SEI [CN02, Nor03] proposes the following 
software engineering practices listed in Table 8-1 as necessary practices supporting an or¬ 
ganization’s capability and technology to create, evolve, and maintain core assets and 
products: 

• Architecture Definition - This practice area defines the activities supporting the 
definition of software architecture for a product line and addresses quality attrib¬ 
utes, system interaction requirements, and organization business goals. The focus 
may be infrastructure focused (e.g., operating system, protocols, middleware), or 
focus on the application functionality. The software architecture^^^ is a major com¬ 
ponent supporting the development of the core assets, affecting how well an organi¬ 
zation fields products form a shared asset repository, supporting explicitly allowed 
variations [CN02, Nor03]. 

• Architecture Evaluation -The architecture of a system, an early design artifact, 
represents one of the earliest design decisions, and perhaps the most difficult to 


No completely accepted definition for software architecture has emerged, although over ninety have been discovered by [BCC+03]. 
[CBB+03]: suggests the following definition:- Software Architecture is the structure or structures of a system, which comprise elements, 
their externally visible properties, and the relationships among them [CBB+03]. 
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change over time. It is at this point that the software arehitects establish the sys¬ 
tem’s quality foundations (e.g., security, reliability, usability). The abstraetion of 
the architeeture at this point supports communication among the system’s key 
stakeholders. Several different arehitecture evaluation teehniques exist including; 
the Arehitecture Tradeoff Analysis Method, the Software Arehitecture Analysis 
Method, aetive design reviews. Software Performance Engineering, and active re¬ 
views for intermediate designs [CN02, Nor03]. 

• Component Development - The term component^®^ like software arehiteeture and 
object has collected many meanings, and not yet standardized. The component 
functionality is normally encapsulated and paekaged with an interface provided to 
other components employing an agreed upon intereonnection method. Components 
form the building blocks for applications, linked by communieation and conneetion 
teehnologies ineluding the Common Objeet Request Broker Architecture 
(CORBA), from the Object Management Group; Distributed Common Objeet 
Model (DCOM), originally from Mierosoft; and JavaBean [DRS99], developed by 
Sun [CN02, Nor03]. 

• COTS Utilization - Commereial-off-the-shelf and non-developmental items (NDI) 
may also be used as eore assets in a product line. In the past deeade, the growth of 
middleware teehnologies supported the employment of COTS / NDI components in 
large-seale systems, introducing a new set of risks, eonstraints, and tradeoffs 
[Nor03]. Key issues for consideration include COTS evaluation process, adaptabil¬ 
ity, and vendor support for COTS produets [BB99, Car99, Voa99, HP99b, 
SWR+99, PHW99, MGOO, ABCOO, CN02]. 

• Mining Existing Assets - Much of today’s software systems evolve as extensions 
of legaey systems. However, legaey software involves resurrected and rehabili¬ 
tated software or serviee in a new system beyond the original design. Mining ex¬ 
pands signifieantly upon the current practice of small-grained reuse of eode, sub¬ 
routines, or small programs and may include a higher-level foeus on the organiza¬ 
tion’s business proeesses and software architeeture, in addition to eonsideration of 
the normal constraints of cost, schedule and functionality. Mining assets are re¬ 
source intensive activities and focus on a wide range of assets besides code, inelud¬ 
ing business models, rule bases, and budgets. Prime candidates include algorithms, 
interfaee specifications, performance models, test plans, and available architeeture 
doeumentation. If quality doeumentation does not exist, reeonstruction of existing 
doeumentation with enhancements such as tradeoff options, presents an alternative 
approaeh supporting improved use of the component [BOSOO, CN02, Nor03]. 

• Requirements Engineering - The IEEE defines a requirement^and [Bro87] 
noted the “hardest single part of building a software system is deciding precisely 
what to build.. .the detailed teehnical requirements” [Bro87]. [BE91, GGR+94, 
GGR+95, Som95, Pre97, EKOO] provide in-depth coverage of requirements engi- 


^ Components are the principal computational elements and data soures that execute m a system [CBB+03]. 
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A requirement is: (1) A condition or capability needed by the user to solve a problem or achieve an objective. (2) A condition or 
capability that must be met or possessed by a system or a system component to satisfy a contract, standard, specification, or other for¬ 
mally imposed documents. (3) A documented representation of a condition or capability as in (1) or (2) [IEE90]. 
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neering techniques supporting systematic and repeatable processes for completely, 
consistently, and relevantly eliciting, analyzing, specifying, verifying, and manag¬ 
ing requirements. Requirements engineering for a product line differs from the re¬ 
quirement process of a single system and includes: capturing the anticipated varia¬ 
tions over the projected future of the product line, finding commonalities and identi¬ 
fying variations, preparing a product-line-wide set of requirements and product - 
specific requirements. It may involve a broader verification process occurring most 
likely in stages, and adaptation to support the dual nature and staged (e.g., common, 
specific) nature of product line requirements engineering [CN02, Nor03]. 

Product line analysis is a requirements process for engineering a product 
line of software-intensive systems [CDK+01]. The [CDK+01] suggested method¬ 
ology encompasses the elicitation, analysis, specification, and verification and vali¬ 
dation of the requirements for a product line. Four interrelated products support the 
product line requirements model for elicitation, analysis, specification, and verifica¬ 
tion: object modeling [RFOl], use-case modeling, feature-modeling, and the dic¬ 
tionary, with the goal to achieve high cohesion / low coupling [CDK+01]. 
[CDK+01] also propose product line analysis methods for establishing the require¬ 
ments for a product line of software-intensive systems supported in the context of 
product line development, assisted by modeling techniques and tools [YMK+00, 
SSP+00, STOO, HOF+00, CN02]. 

[CDK+01] also show how to build a requirements model from work prod¬ 
ucts, based on object modeling, use-case modeling, and feature-modeling tech¬ 
niques. Requirements are statements of what a system must do, how the system 
must behave, the properties it must exhibit, the qualities it must possess, and the 
constraints that the system and its development must satisfy. A feature is a distinct 
aspect, quality, or characteristic of a software system or systems visible to the user. 
Strategies and methods for product line requirements modeling [KNOO, CJBOO, 
AOV+00] include the feature-driven strategy and a use-case-driven strategy. In a 
feature-driven approach, requirement modeling focuses on the features, while de¬ 
velopers use the use-case strategy to discover requirements [CDK+01]. 

• Software System Integration - This practice involves combining individual soft¬ 
ware components into an integrated whole in a process where components are inte¬ 
grated into subsystems or when subsystems are combined into products, either dis¬ 
cretely supporting a waterfall approach, or continuously supporting an incremental 
methodology. A key point about product line integration is that cost of integration 
identified in the product line scope, core assets, and production plan for the archi¬ 
tecture’s planned life cycle is amortized over many products, versus a single system 
integration. Once the components and interfaces have been tested and verified there 
should be very little effort needed for new variations and adaptations. Specific 
practices supporting component integration includes patterns [Fow97], object tech¬ 
nology, wrapping for recovery or discovery solutions, and middleware [CN02, 
Nor03]. 

• Testing - Testing performs two primary functions described in detailed by [BL91, 
Som95, Pre97, Roa98, and LKOO]. First, testing continually supports developers 


197 



ability to identify faults leading to failure in the development phase, and seeondly, 
at a later time when testers determine whether a system ean perform to meet its re¬ 
quirements [CN02, Nor03]. Testing is also a eritieal part of the V&V and quality 
proeesses. Metrics and measurements for testing and reliability supporting credibil¬ 
ity in simulation software include [Ber94, CFG+94, Jon95, FV96, PGF96, Sta96, 
MV96, JA97, Wie97b, AVL+97, Sch99b]. 

• Understanding Relevant Domains - Domain understanding evolved as a major fac¬ 

tor in this research. Domains are areas of expertise and domain knowledge avail¬ 
able for creating future systems or set of systems, with an understanding that 
knowledge from several domains is normally required to build a single product. 
Understanding a specific domain normally entails the identification of areas of ex¬ 
pertise, identification of recurring domain problems and known solutions, capturing 
and representing this information to stakeholders, for the duration of the effort. 
Domain comprehension supports understanding the commonality and variability of 
potential future product within the scope of the product line [CN02, Nor03]. 

b. Technical Management Practice Area 

Technical management practices are those management practices necessary 
to engineer, develop, evolve, and maintain to core assets and the products and encompass: 


• Configuration Management - Software configuration management (CM) [Bro98, 
CN02, Nor03, BCC+Olb] includes the following activities: software configuration 
identification, software configuration control, software configuration status ac¬ 
counting, software configuration auditing, and software release. Configuration 
management processes matured steadily since the early 1970s, and include new or 
reengineered methods, management techniques [Bro96, BL91, Som95, Pre97, 
LKOO], technology [STS94a], and products [BH99] continuously evolve [Bro87]. 

However, CM for a product line is complex. The core assets and each of the 
products in the product line constitute a configuration to manage, and the manage¬ 
ment of all configurations needs coordination under one process. We will compare 
and contrast the CM requirements for a single system with the CM requirements for 
a product line. First, a single system manages each version’s configuration; a prod¬ 
uct line requires CM for each version of each product. A single-system CM process 
may separately manage each product and all its versions; however, in a product line 
system the core assets integrated across all products require a single, unified CM 
process. Third, in a single system, the component developers and product develop¬ 
ers are often the same, whereas in a product line must support the CM of the core 
assets, normally developed by one team and supporting parallel production by sev¬ 
eral other teams [CN02, Nor03]. The additional requirements levied by a product 
line CM suggest the need for disciplined CM techniques and processes, supported 
by robust CM tools. 
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Additional new CM techniques include the uniform version management 
framework, explained by [WMC02], which supports the definition of version mod¬ 
els and addresses the orthogonal differences between the version and data models; a 
test bed model that separates CM repositories from CM policies detailed by 
[HCH-l-02]; visualization techniques [EGF02], and software merging version con¬ 
trol [Men02]. [AB02] describes open source software projects CM methods, while 
[SJW+02] discusses open source software maintainability. [Kil02] cites the follow¬ 
ing benefits to software accuracy attributable to good software configuration man¬ 
agement: longer shelf life of V&V work, confidence that model results are consis¬ 
tent, lower likelihood of undetected errors in the code, less experimentation and 
training needed to operate the model, and CM provides a venue to pool resources 
for model changes, including legacy models with multiple users [Kil02]. 

Configuration management has proven inconsistent in the Department’s 
M&S domain. Citing the restrictions of the current W&A directives, [Cau95] is 
concerned that the process is so complex, time-consuming and expensive that 
changes are often made to the M&S before the V&V process is completed, poten¬ 
tially invalidating the process. [Mue97a] described the Susceptibility Model As¬ 
sessment and Range Test (SMART) project, which integrated configuration man¬ 
agement with V&V processes to improve M&S credibility. Decision-makers trust 
in an M&S tool requires strict configuration management, yet the M&S analytical 
tool set must have the flexibility and depth to resolve complex problems. 

• Process Definition - The process definition practice area involves an organization’s 
capability to define and follow documented processes. Product line management 
requires the disciplined interaction of separate organizational entities adhering to 
mature processes. Each core asset has an attached process explaining how to em¬ 
ploy the core asset to build products in a product line [CN02, Nor03]. Software 
process modeling supports process definition, describing the abstract description 
and models of important defined software development processes executed by a 
human or a machine to achieve the following goals: 1) facilitate human understand¬ 
ing, 2) support process management, and 3) support process improvement.. 

• Product Line Scoping - Scoping bounds a system or set of systems defining behav¬ 
iors, characteristics or aspects included or excluded from the product line formal¬ 
ized in a scope definition document supporting the requirements engineering proc¬ 
ess, or influencing the market analysis for a product line variant [CN02, Nor03]. 

• Technical Planning - The product line requires no new planning processes, how¬ 
ever, core asset development, core asset maintenance, core asset production, and 
core asset reuse plans are unique product line endeavors [CN02, Nor03].. 

• Technical Risk Management - Risk management is the practice of managing risks 
within a project, organization or a group of organizations. Risk management is a 
critical process for a product line since the risks involve more than one product, 
with potentially far-reaching consequences [CN02, Nor03]. 

• Tool Support - A product line requires tools to support concurrent development, 
employment and maintenance of multiple core asset artifacts. Most likely several 
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interoperable eomputer-aided software engineering tools will manage the product 
line [CN02, Nor03]. 

• Data Collection, Metrics, and Tracking - Measurements and metrics support and 
guide management decisions about whether organization goals and objectives are 
met over time [CN02, Nor03]. Product line systems are managed in many ways 
just like a single product system, except for the additional requirements to track the 
core asset development, product development, and overall management of the 
product line. [Ber94, CFG+94, Jon95, FV96, PGF96, Sta96, MV96, JA97, 

Wie97b, AVL+97, Sch99b, and lEEOOl] provide guidance on measurement and 
metrics. 

• Make/Buy /Mine /Commission Analysis - Normally a product may be built in- 
house, purchased from a commercial company, commissioned for development, or 
mined from in-house assets based on a core asset development decision process in¬ 
cluding quality, cost, product line requirements, architecture, variation flexibility, 
maintainability, and schedule [BSW+99, CN02, Nor03]. The product development 
activity depends upon the product line scope, core assets, production plan, and the 
requirements for the individual products. Product line organizations are flexible 
and may include a product group for several products, or one product group per 
product. Management according to [WapOO, VWOO, TCOOO] also makes assets 
available for reuse, retains responsibility for success or failure, manages external in¬ 
terfaces, creates an adoption plan, and acts as or empowers the product line cham¬ 
pion. Management support required changes to cultural perspectives [Wie96b] and 
allows new products to align with existing assets, or update the assets to meet the 
new requirements. 

• 

c. Organizational Management Practice Area 

Organizational management practices are those management practices nec¬ 
essary to orchestrate the entire core assets and products line effort and include: 


• Building and Communicating a Business Case - There are several generally ac¬ 
cepted costing approaches for building and communicating a business case, includ¬ 
ing top-down, bottoms-up, algorithmic, analogy, and expert judgment [BBW+00, 
CN02]. A significant body of work exists on cost estimation approaches based on 
project characteristics. Software estimation tools include the non-proprietary Con¬ 
structive Cost Model models first introduced by Barry Boehm COCOMO model in 
1981, the Revised Intermediate COCOMO (REVIC) model, Putnam’s Software 
Eifecycle Model (SEIM) and Boehm’s COCOMO II; in addition to software size 
estimation tools such as SEER-SSM discussed and compared by [BNT93, STS93a, 
STS93b, STS94b, McG97, BA99, and NogOO]. [SK02] advocates the development 
of newer empirical cost and schedule estimation approaches based on the increased 
use of COTS, application generators, simulations, and fourth generation languages 
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today, while [Wit96] proposes an investment analysis tool for software produet line 
business ease development. 

• Customer Interface Management - Managing the customer interface for a product 
line differs significantly from a single-product organization, requiring new methods 
for managing customer expectations, negotiating requirements, developing, evolv¬ 
ing, and maintaining customer products and providing customer support [CN002, 
Nor03]. 

• Developing and Implementing an Acquisition Strategy- This process supports the 
acquisition of products and services by contract, such as those organizations pur¬ 
chasing or commissioning products, and especially important for government agen¬ 
cies, such as the Department. Acquisition of product lines are typically structured 
differently with the acquisition of the core assets versus a product as a contract de¬ 
liverable, although the role of the architecture in a product line may provide oppor¬ 
tunities for contracting flexibility [CN02, Nor03]. Product lines present several 
challenges to the Department [Jon99], requiring the development of Department 
product line acquisition processes [BFJ99, BFG+00, Cam02, BOS02], and the shar¬ 
ing of lessons-learned from successful Department product line initiatives [CDS02]. 

• Funding - A product line requires significant up-front investment to build or ac¬ 
quire the core asset base, complete initial analysis efforts, and establish the produc¬ 
tion infrastructure; then evolving and sustaining the effort [CN02, Nor03]. This in¬ 
dicates a need for a strategic plan and stable, sustained funding, a significant chal¬ 
lenge for product lines in the Department. 

• Launching and Institutionalizing a Product Line - Launching and institutionaliz¬ 
ing a product line involves the initiation and improvement of the product line prac¬ 
tices appropriate for a given organization, and viewed as a practice area for apply¬ 
ing the other practice areas [CN02, Nor03]. 

• Market Analysis - A market analysis is an early step for a product line, establishes 
the product line scope, providing analysis on the possible product line commonality 
and variability [CN02, Nor03]. 

• Operations - Operations define which part of the organization develops the core as¬ 
sets and products, and directs planning, processes, strategies, policies, and con¬ 
straints [Nor03]. Management provides the resources, coordination, and supervi¬ 
sion, critical to success, ensuring coordination of the operations and communica¬ 
tions activities of the product line effort with a concept of operations (CONOPS). 

• Organization Planning - The product line planning process, as mentioned earlier, 
is not unique, but does require product line adoption plans, core asset funding plans, 
and due to its importance, organization-level configuration management plans 
[CN02,Nor03]. 

• Organizational Risk Management - Organizational risk management involves risk 
management at the strategic level, and requires a great deal of coordination across 
project boundaries supported by open communication, integrated management, 
teamwork, a forward-looking view, global perspective, and shared product vision 
[CN02,Nor03]. 
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• Structuring the Organization - A product line approach requires new roles and re¬ 
sponsibilities supporting core asset development and product development from the 
core asset base. An early decision identifies where to locate the developers and 
maintainers of the product lines core assets and the organizations that build the 
product lines. Typically, the developers and maintainers may be located separately 
from the product line builders or they may be located together [CN02, Nor03]. 

• Technology Forecasting - Technology forecasts for product lines include two fo¬ 
cus areas, internal development for tools, processes, or methods; or customer solu¬ 
tions such as technology or solutions that may possibly affect product line features 
or capabilities [CN02, Nor03]. 

• Training - Training is a core activity of organizations developing software and 
supports the initial product line adoption and the long-term product evolution; and 
requires management’s commitment, an effective training plan, and support of 
product line adoption process or process improvement [CN02, Nor03]. 

D. EVOLVING SOFTWARE ARCHITECTURE THEORY 
1. The Federal Enterprise Architecture (EEA) 

The Office of Management and Budget recently initiated a Federal Enterprise Ar¬ 
chitecture (FEA) program [OMB02, FEA02a, FEA02b, SAG02, Hay03b, BBT-l-03] sup¬ 
porting the President’s e-Gov guidance, focuses on information technology investments, 
and designed to facilitate cross-agency analysis and improvement. The lack of architecture 
in the Federal Government is a systemic management weakness repeatedly cited since the 
early 1990s [BBT-l-03]. The Federal Enterprise Architecture’s four-layer segmented 
structure [FEA02a] include systematically derived and captured structural descriptions in 
useful documentation (e.g., models, diagrams, narrative) for a given enterprise, including a 
single organization, or functional area, or a mission area that transcends more than one or¬ 
ganizational boundary [BBT-l-03]. The existing agency-specific architectures will serve as 
the foundation for the FEA, with five reference models or views under development to as¬ 
sist with aligning existing data with the FEA, 

• The Business Reference Model describes Federal Government business operations, 
independent of individual agency implementation, 

• The Performance Reference Model provides a common set of general performance 
outputs and measures for agencies. 


The four layers of the FEA are the Technology, Application, Data, and Business Architectures [FEA02a]. 
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• The Data and Information Reference Model describes at an aggregate level, the data 
types supporting agency operations, and the relationships among the data, 

• The Service Component Reference Model identifies and classifies information tech¬ 
nology service components, and promotes reuse, 

• The Technical Reference Model describes how technology supports delivery of ser¬ 
vice components and includes relevant standards [BBT+03]. 

The Department, as an agency of the Federal Government develops software-intensive sys¬ 
tems with the intent of achieving interoperability with other stakeholders, including other 
Government agencies (OGA), will support the FEA program. 

The FEA suggests the use of Elnified Modeling Eanguage (LIME) [E1ME99] for de¬ 
fining and applying data models and standards [EEA02a]. In addition, XME, emerging as a 
government and industry standard, provides a recommended foundation as the default for¬ 
mat for moving and sharing highly structured information, as well as less highly structured 
information between E-Gov data architectures [KJOl, EEOl, EEA02a]. Data interoperabil¬ 
ity principles for the EEA Data Architecture support improved interoperability by, 

• Avoiding non-standard data syntaxes, 

• Seeking industry vocabularies before developing custom schemas, 

• Avoiding the “one size fits all” schema concept, 

• Registering the semantics of shared data elements, 

• Documenting service interfaces in a standard way [EEA02a]. 

2. Emerging Software Architecture Concepts 

The software architecture discipline is relatively new, and has not been completely 
defined and applied consistently to the life-cycle management of software-intensive sys¬ 
tems. Much of the current software architecture research stems from the earlier works of 
[Par72, Par76, Par79, PW92, SG93, and SC96] and their observations that software con¬ 
sists of many structures, and that a system is a collection of related parts. Although there is 
not currently agreement on a precise definition of a system’s architecture , the IEEE Rec¬ 
ommended Practice for Architectural Description of Software-Intensive Systems [lEEOOb], 
cites a consensus on the use of multiple views^^^, reusable specifications [BS95, Gac95, 
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Architecture is the fundamental organization of a system, embodied in its components, their relationship to each other and the envi¬ 
ronment, and the principles governing its design and evolution [lEEOOb] 
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a) A view is a representation of a whole system from the perspective of a related set of concerns [lEEOOb]. b) A view is a representa¬ 
tion of a set of system elements and the relationships associated with them [CBB+03]. 

203 



HBL97, MR02] for models within views, and the relationship of arehiteeture to the system 
eontext. The foundation of the approved [lEEOOb] builds on the following concepts and 
relationships, 

• Every software-intensive system has an architecture, however, an architecture is not 
a system, 

• An architecture and an architectural description are not the same, 

• Architecture standards, descriptions, and development processes can differ and be 
separately developed, 

• Architecture descriptions are inherently multi-viewed, 

• Effective architecture description standards support the separation of an object’s 
view from its specification [MEHOl]. 

The architecture of a system^^"^ is a critical component supporting the engineering 
process and the life cycle modeP^^ of the system [EB95, SNH95, MTW96, CN96, Eow97, 
HBE97, EB98, CWK99, BBG+00, BosOO, lEEOOb, HHPOO, MEHOl, MMOl, EG02, 
EEC+02, MR02, SB02b, Era03, CBB+03, GA03, SPG03, Tol03]. As large-scale, legacy 
software-intensive systems evolved, the initial architectural emphasis was on the hardware 
and network architecture components of the information systems, until the complexity of 
the software technology and the cost of software development necessitated a change in the 
emphasis to include the software-related architecture issues [ShaOl] of today’s software¬ 
intensive system. Software architecture especially for large, software-intensive systems, 

• Serves as the system blueprint, a focal point for the project development team and 
mutual communication, 

• Serves as the foundation for the system’s quality attributes, 

• Provides the first artifact for early design decisions indicating the system meets 
requirements, 

• Supports a transferable abstraction of the system for activities including post¬ 
deployment maintenance or mining [CN96, BBG+00, BOSOO, MR02]. 

The software architecture is key to the success of any software project [EB98, 
BCK98] and critical to the success of a product line initiatives [Jon99, BosOO, DSOO, 

ProOO, ShaOOa, MorOOa, DonOO, CN02]. Architectural drivers, influencing the entire life 


System is defined as a collection of components organized to accomplish a specific function or set of functions [lEEOOb]. 
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The life cycle model is a framework containing the processes, activities, and tasks involved in the development, operation, and main¬ 
tenance of a software product, which spans the life of the system from the definition of its requirements to the termination of its use 
[lEEOOb]. 
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cycle of a system, inelude quality attributes, business goals, and system interactions 
[MR02]. Product line arehiteetures also inelude the allowable product variability and the 
different instantiations of the product, since these too are produets. 

There may also be many views in a software architeeture domain, whieh show spe- 
eifie properties of the software system [lEEOOb]. These different architecture views ad¬ 
dress diverse issues encountered with the design process, ineluding the logieal view, proe- 
ess view, physical view, and the development view. Other terms of references for software 
arehiteeture design views include behavioral, functional, structural, and data modeling 
views, however, the key point made by [BBOO] is that software architecture design is the 
product of a multi-dimensioned process composed of independent and orthogonal views, 
impacted by variability, either planned or unintended. 

[CBB+OS] defines an architectural style (e.g., pattern) as a specialization of ele¬ 
ment and relation types, together with a set of eonstraints on their use. In this researeh ef¬ 
fort, an arehitectural style consists of a set of constraints on the arehiteeture, which define 
the set or family of architectures, and satisfy them with a number of major styles. These 
styles may include general struetures of pipes and filters [SC96], layers [BBG+00], black- 
boards, object-orientation, and implicit invocation [BosOO]; distributed systems; interac¬ 
tive systems; adaptable systems; and other styles including batch, interpreters, proeess con¬ 
trol, specification-based, and rule-based. 

Software arehiteeture theory is a rapidly evolving component of software engineer¬ 
ing with a wide range of diverse concepts, styles, and views including; the development of 
domain speeific repository for components [Gae95, HBE97], structural modeling [CB96], 
software arehiteeture overview [CN96], transitioning to a model-based engineering arehi¬ 
tectural style [GP96], and ineludes industrial best-praetiees for evaluating software arehi- 
tectures [ABC+97]. The evolving software architecture theory body of knowledge ineludes 
a wide spectrum of topics: evaluating the quality attributes of a software architeeture 
[BKW97], reeonstructing a software arehiteeture with automated support [KC97], reusable 


Shaw and Clements define architectural styles as a set of design rules that identify the kinds of components and connectors that may 

be used to compose a system or subsystem, together with local or global constraints on the way the composition is done [SC96]. 
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Mary Shaw and David Garlan introduced implicit invocation in (1996) as a style that organizes the system in terms of components 
that generate events, possibly containing data, and that consume events. An example is the JavaBeans standard [BosOO]. 
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components implemented via a relational hydrograph model [Luq90, HBL97, LG97], soft¬ 
ware arehitectural transformation via automated code transformation [CWK99], interop¬ 
erability [Sut99], attribute-based arehitectural styles [KK99], and methods for documenting 
architectural layers [BBG+00]. 

3. Software Architecture Views 
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A system is a colleetion of components organized to aeeomplish a specifie func¬ 
tion or set of functions [lEEOOb]. In addition to defining arehiteeture, [PW92] emphasized 
eertain arehitectural considerations for different stakeholders or for different uses with a 
variety of system views. A sample of other reeent advanees in software arehiteetures noted 
in Table 8-2 include [Kru95] describing four views of software arehiteeture for system 
building; the eollaborative work of [SNH95] who identified four additional industrial use 
views; four business views [HSOOe], the development of viewpoints^'^ [lEEOOb] to desig¬ 
nate the means used to eonstruct individual views; and the three eategories of views identi¬ 
fied as viewtypes^^'^ [CBB+03]. 
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Table 8-2. Software Arehitectural Views / Viewpoints / Viewtypes (Erom [CBB+03]) 


A system also includes individual applications, traditional systems, subsystems, systems of systems, product lines, product families, 
whole enterprises, and other aggregated types [lEEOOb]. 
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A viewpoint is specification of the conventions for constructing and using a view. A pattern or template from which to develop 

individual views by establishing the purposes and audience for a view and the techniques for its creation and analysis [lEEOOb]. 
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A viewtype defines the element types and relationship types used to describe the architecture of a software system from a particular 
perspective [CBB+03]. 
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There have been many attempts to overeome the formidable risks and difficulties 
experienced in the design, development, deployment, and evolution of software-intensive 
systems, with improved software engineering practices, procedures, and techniques 
[MR02]. In 1996, the IEEE chartered the Architecture Working Group (AWG) to imple¬ 
ment the approved recommendations from the 1995 IEEE Architecture Planning Group for 
software-intensive systems. The [lEEOOb] charter tasked the AWG to define terms, princi¬ 
ples and guidelines for the consistent application of architectural precepts for systems 
throughout the entire life cycle. 

The AWG elaborated on architectural precepts and potential benefits for software 
products, systems, and aggregated systems (e.g., systems of systems); provided a frame¬ 
work for architectural attributes; and developed a roadmap for architectural precepts in the 
generation, revision and application of IEEE standards. In developing [lEEOOb], the AWG 
intended to capture the architectural information contained in the various products of the 
system development process illustrated in Eigure 8-1 and devise an architectural descrip¬ 
tion^^ 

• Expressing the system and its evolution, 

• Supporting communication among the system stakeholders, 

• Allowing a consistent comparison of architectures, 

• Supporting system development, 

• Identifying the system’s persistent characteristics and supporting principles for fu¬ 
ture changes, 

• Verifying the system’s implementation as compliant with an architectural descrip¬ 
tion, 

• Recording contributions to the body of knowledge of software-intensive system ar¬ 
chitectures [lEEOOb]. 


An architectural description is a collection of products to document an architecture [lEEOOb]. 
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Figure 8-1. Model of an Architectural Description (From [lEEOOb]) 

4, Software Architecture Quality Analysis Methods 

[DN02] suggest that over the past several years, software architecture emerged as 
the appropriate level for addressing software quality, and recent efforts to understand the 
design patterns and architectural styles contribute to that quality analysis effort. In their 
research [DN02] provide a concise and recommended survey on eight representative soft¬ 
ware analysis methods in different domains, compare and contrast similarities and differ¬ 
ences through study, comparison, and classification, and offer guidelines supporting the use 
of the most suitable method for an architecture assessment process. 

[DN02] fit these eight methods into three categories or communities: software met¬ 
rics, scenario-based, and attribute model-based analysis. The software metrics analysis 
technique uses module coupling and cohesion theories or more abstract evaluations to de¬ 
fine predictive measures, of quality; while scenario-based methods applied over the past 
several years have a maturity and a certain level of face validation based on use; while the 

attribute model-based is too new to evaluate [DN02]. These eight methods include: 
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• Scenario-Based Arehiteeture Analysis Method (SAAM) supports a general under¬ 
standing of the general arehitectural eoneepts supporting proof that the software 
system meets more than the funetional requirements, 

• SAAM Founded on Complex Seenarios (SAAMCS) eonsiders that the eomplexity 
of seenarios is the most important faetor for risk assessment, 

• Extending SAAM by Integration in the Domain (ESAAMI) a eombination of ana- 
lytieal and reuse eoneepts, integrates SAAM in the domain-specifie and reuse-based 
development proeess, eonsidering only the problem deseription, requirements 
statement, and arehiteeture deseription, 

• Software Arehiteeture Analysis Method for Evolution and Reuse (SAAMER), an 
extension of SAAM assessed quality objeetives based on two attributes, evolution 
and reusability, 

• The Arehiteeture Trade-off Method (ATAM) an attribute-based model, provides a 
way of understanding a software arehiteeture’s eapability with multiple, eompeting 
quality attributes, 

• Seenario-Based Arehiteeture Reengineering (SBAR) evaluates multiple quality at¬ 
tributes of the arehiteeture design in a scenario-based review of the software quali¬ 
ties of a specifie software arehiteeture or system, 

• Arehiteeture Eevel Prediction of Software Maintenanee (ALPSM) analyzes main¬ 
tainability of a software system employing scenarios at the software arehiteeture 
level and using the size of the ehange as a predietor, 

• Software Arehiteeture Evaluation Model (SAEM), a quality attribute model, estab¬ 
lishes the basis for the software arehiteeture internal and external quality evaluation 
and predietion of final system quality [DN02]. 

Software quality is one of three dependent, user-oriented produet eharaeteristics, in 
addition to eost and sehedule [BKW97]. A proeess mature organization may predict and 
control cost, however, process maturity does not automatieally translate into product qual¬ 
ity, which requires mature teehnology to predict and control quality attributes [BKW97]. 
[BKW97, BKBOO, BKBOl, and BBK02] deseribe the quality attribute requirements for 
performanee, seeurity, modifiability, reliability, and usability and the signifieant infiuenee 
of these attributes for evaluating software arehiteetures. 

Other representative software arehiteeture quality assessment teehniques inelude the 
Model-Driven Arehiteeture Theory [SieOl, Era03], supported by the Objeet Management 
Group, simulation, mathematieal modeling, and experienced-based assessment teehniques 
[BosOO]. [CBB+OS] identified Department software arehiteeture theory as a fledgling prae- 
tiee and eonsiders the JTA and C4ISR arehiteeture framework focus on the system arehi- 
tecture, and HLA publish-subseribe meehanism lacking in software arehiteeture substanee. 
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However, an understanding of software arehitecture in the M&S domain continues to grow, 
and [T 0 IO 2 , Tol03] discusses future architecture requirements for HLA development. The 
SEI also contributed the Attribute-Based Architecture Style (ABAS) [KK99], Architecture 
Based Design (ABD) [BK99] and Attribute Driven Design (ADD) [BKBOO] methods. 

The Architecture Based Design (ABD) method [BK99, BBC+00, BBKOO] provides 
a recursive framework with three foundations of attributes (e.g., functional, quality, and 
business requirements) at a level of abstraction supporting the necessary variation for pro¬ 
ducing products, and includes function decomposition, architectural styles, and software 
templates for designing high-level software architectures. [BKBOl] develops functional, 
quality and business requirements, or architectural drivers, at a level of abstraction that al¬ 
lows for the variation needed to produce specific products for a product line or any type of 
system with a long lifeline. The ABD method [BK99, BBC+00, BBKOO] provides a proc¬ 
ess for designing the conceptual software architecture, support for organization functions, 
identification of synchronization points, and allocation of functions to processes, conclud¬ 
ing with allocation commitments to classes, processes or operating systems. A product of 
the ABD process [BBC+00] is a collection of software templates , which constrain the 
implementation of different types of components with a description of component interac¬ 
tions and responsibilities. 

Another SEI-sponsored approach for defining a software architecture based on a de¬ 
sign process driven with specified functional and quality attribute requirements possessed 
by the software is the Attribute Driven Design (ADD) [BKBOO, BKBOl, BBK02] method, 
a recursive, decomposition process where at each stage of the decomposition process, cho- 
sen attribute primitives [BKBOO] attempt to satisfy a set of quality scenarios [BKBOl]. 

In addition the development of quality architectures necessitates early consideration of fac¬ 
tors affecting the various quality attributes and the impact on software design [BKBOO, 
BBK02]. [BKBOO, BKBOl, and BBK02] further suggest the use of general scenarios to 
support development of quality attributes, with each general scenario consisting of: 
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A software template defines the software element of a particular type, including patterns describing interactions with shared services 

and infrastructure, and citizenship responsibilities [BBC+00]. 
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An attribute primitive is a collection of components and connectors that 1) collaborate to achieve some quality attribute goal, nor¬ 
mally expressed in a general scenario; and 2) is minimal with respect to the achievement of these goals [BKBOl]. 
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• The stimuli requiring an architeeture’s response, 

• The souree of the stimuli, 

• The eontext in which the stimuli occurs, 

• The type of system elements involved in the response, 

• Possible responses, 

• The measure’s employed to characterize the architecture’s response [BKBOO, 

BKB01,BBK02]. 

An example of a reliability general scenario is the case where an internal component fails. 
The system recognizes a failure of an internal component and possesses capabilities to 
compensate for the fault. The SEI developed quality attribute workshops as a method to 
analyze a system against a number of critical quality attributes [BW02, BEL+02]. 

Quality attribute design primitives are basically templates and provide building 
blocks for developing architecture designs supporting the achievement of specific quality 
attribute goals [BKBOl]. An attribute primitives addresses one or more quality attributes 
characterized by one or more general scenarios [BKBOl]. In a product line different prod¬ 
ucts may have different quality attribute requirements or the products may exist simultane¬ 
ously and only vary in terms of different attributes, characteristics, or scale factors. 

5, The extensible Markup Language (XML) 

The World Wide Web Consortium (W3C) developed the key technical standards for 
XML, including Namespaces the XML Information Set, and XML Inclusions [Sal02]. 
XML, as an application profile or restricted form of the Standard Generalized Markup 
Language (SGML) describes a class of data objects called XML documents and partially 
describes the behavior of programs which process them [Sal02]. Storage units called enti¬ 
ties, which contain parsed or unparsed data, comprise XML documents. Parsed data in¬ 
cludes characters, which form either character data or markup. Markup encodes a descrip¬ 
tion of the document’s storage layout and logical structure [Sal02]. 

Expanding areas of XML research include W3C working group efforts for an XML 
Query (XQuery) data model [CLM-l-03], and query language [PMM-l-03]. XML application 
areas include: database interoperability [HinOl], heterogeneous software systems [YouOl, 
TAOl], XML schema integration [HalOlb], real-time system data interchange [PraOl], 
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common data attributes [ZobOl], an XML technology assessment [BerOl], and workflow 
and document management [ACD+02]. 

In the M&S area, XML uses include: data interchanges for equipment and perform¬ 
ance information [LKBOl, LPA+02], support of command and control communications 
requirements [SBOO], data management and architecture description [SB02b], the devel¬ 
opment of integration and collaborative toolsets [GRG+02], support to major simulation 
development efforts [And02], and scenario generation [RK02]. Web-based M&S tech¬ 
niques employing XML for development and interoperability have also emerged [BZP+02, 
Hob03, Tol03]. 

However, [MCF+02] identify challenges in realizing XML’s full potential including 
risks that data will lack definition, incompatible data definitions, and the proliferation of 
vocabularies and structures. [BerOl] suggest that the lack of agreement on data representa¬ 
tions and conceptual data models represent a major obstacle to data interchange among leg¬ 
acy systems, but concludes that: 

Data interoperability is feasible without requiring a comprehensive data 
standard, and that methods for incrementally growing localized standards 
and bridging the gaps among them without requiring global agreement ap¬ 
pear possible. Further assessments are proposed to determine the practical 
feasibility of applying this approach to DoD operations [BerOl]. 

6, Software Architecture Description Languages (ADL) 

Various Architectural Description Languages (ADL) support current software ar¬ 
chitecture research and development efforts with the potential for employing common 
coarser-grained architectural elements (e.g., components and connectors) and interconnec¬ 
tivity schemes for architecture-based development, and featuring formal modeling nota¬ 
tions, analysis, and development tools operating on architectural specifications [MTOO]. 
However, the research community currently lacks consensus on ADLs, the capabilities ex¬ 
pected from an ADL toolset, and what aspects of a software architecture to model. [MTOO, 
CBB+03] suggest that no existing ADL tool provides the complete capability to document 
software architectures, although many ADLs perform well in specific areas such as concep¬ 
tual frameworks, concrete syntax, parsing, displaying, compiling, analyzing, or simulating 
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architectural descriptions in specific languages. While there is no generally accepted defi- 

nition of an ADL, [MTOO] proposes a definition and classification for ADLs: 

An ADL must explicitly model components, connectors, and their configu¬ 
rations; furthermore, to be truly usable and useful, it must provide tool sup¬ 
port for architecture-based development and evolution. These four elements 
of an ADL are further broken down into constituent parts [MTOO]. 

Although lacking a consistent definition, [MTOO, CBB+OO] cite several do¬ 
main-specific and general purpose ADLs including ACME, Aesop, Darwin, MetaH, 
Rapide, SADL, UniCon, and Wright. Emphasizing the lack of a standard definition 
and scope, [MTOO] describe ACME as an architectural interchange language ena¬ 
bling integration of support tools across ADLs, while [CBB+OO] categorize ACME 
as an ADL. [MTOO] provide a succinct ADL classification and comparison frame¬ 
work of key ADL properties, identifying capabilities and deficiencies. 

E. SUMMARY 

Chapter VIII discussed traditional product lines, and introduces software product 
lines and software architectures. The chapter also reviews core asset development, reverse 
engineering to develop core assets, reengineering core assets, component development sup¬ 
porting a product line methodology, product line practice areas, and an overview of 
evolving software architecture theory potentially applicable to improved M&S credibility 
and reducing heterogeneous system representation anomalies. 

Product lines and product line practices are new to software engineering, the De¬ 
partment, and the Department M&S domain. Product line practice areas include software 
engineering, technical management and organizational management, which may be appli¬ 
cable to meet some or all of the technical, cultural, and managerial challenges collectively 
hindering the development of improved Department M&S credibility. 

A product line approach to developing and deploying software-intensive systems 
offers great promise for delivering higher quality systems in a shorter time and at reduced 
cost. A product line approach to software-intensive systems can save money and result in a 
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Another definition of an ADL is a language (graphical, textual, or both) for describing a software system in terms of its architectural 
element and the relationships among them [CBB+OO]. 
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faster time to field quality systems. A number of organizations gained order-of-magnitude 
improvements in efficieney, productivity, and quality through a product line approach. 
Product line practice also enables an organization to get its product to the market more rap¬ 
idly. As an example, [BC96] provides a case study of a successful product line implemen¬ 
tation by the CelsiusTech Systems AB of Sweden in the area of large, embedded real-time 
shipboard command and control systems. 

Any software product line implementation strategy in the Department must take 
into account the Department is an acquirer of software systems and not the software devel¬ 
oper, and factor in product line practice areas supporting the Department’s acquisition 
process. Initial efforts to employ product lines in the Department’s M&S domain have 
been limited. 

Components play a major role in many software-intensive systems. There are sev¬ 
eral motivations for a components-based software approach to software: providing a basis 
for commerce in reusable software to meet demand, facilitating the development of flexible 
systems, and reducing the time to design, implement, and deploy systems. As an indication 
of the need for software components at the United States Federal Government level, the 
Office of Management and Budget recently established the Federal Enterprise Architecture 
Program Management Office to remedy the lack of a Federal Enterprise Architecture 
(EEA), a major barrier to the success of the E-Govemment initiative. Currently there is not 
an established basis for how well component models and frameworks contribute to achiev¬ 
ing the desired quality; nor is there any basis for assessing the quality of software compo¬ 
nent interfaces, composition, and component certification. Consensus on the many solu¬ 
tions required for contentious component / composition issues do not appear on the near- 
term horizon. 

A software product line practice area is a body of work or a collection of activities, 
which an organization must successfully master to carry out the essential work of a soft¬ 
ware product line. Many, if not the majority of the practice areas describe activities, which 
are critical to the success of any software project, and provide starting points for organiza¬ 
tions to initiate and master activities and measure progress in adopting a product line ap¬ 
proach. The practice areas in the framework divide into three categories: 1) software engi- 
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neering practices, 2) technical management praetiees, and 3) organizational management 
praetices illustrated in Table 8-1. 

Implementing software produet lines offers potential gains, and entails signifieant 
risks, including technical, organizational, and management issues. [Nor03] explains that 
building and aequiring a software produet line requires diseiplined engineering supported 
by mature teehnieal and organization management proeesses of universal essential aetivi- 
ties and praetiees, which the SEI developed into an online framework , a web-based 
doeument deseribing the competeneies needed to develop and field a sueeessful a software 
product line or any software-intensive system. Based on researeh to date, the SEI proposes 
the software engineering praetiees listed in Table 8-1 as necessary praetiees supporting an 
organization’s capability and technology to create, evolve, and maintain eore assets and 
produets. 

Software arehiteeture theory is a rapidly evolving eomponent of software engineer¬ 
ing with a wide range of diverse coneepts, styles, and views. The evolving software archi¬ 
tecture theory body of knowledge ineludes a wide speetrum of topies: evaluating the qual¬ 
ity attributes of a software arehiteeture, reeonstructing a software arehiteeture with auto¬ 
mated support, reusable eomponents, software arehiteetural transformation, interoperabil¬ 
ity, attribute-based arehiteetural styles, new XME architecture-foeused teehniques and ap- 
plieations, emerging arehiteeture deseription languages, and methods for doeumenting ar- 
chiteetural layers. 

The software arehiteeture is key to the success of any software projeet and eritical 
to the sueeess of a produet line initiatives. Arehiteetural drivers, influeneing the entire life 
cycle of a system, inelude quality attributes, business goals, and system interactions. Prod¬ 
uct line arehiteetures also include the allowable produet variability and the different instan¬ 
tiations of the produet, sinee these too are produets. There may also be many views in a 
software arehiteeture domain, whieh show speeifie properties of the software system. 

These different arehiteeture views address diverse issues encountered with the design proe- 
ess, ineluding the logieal view, proeess view, physieal view, and the development view. 
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Framework in this context suggests a conceptual index, a frame of reference for the information essential for success with product 
lines [Nor03]. 
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Other terms of references for software architecture design views include behavioral, func¬ 
tional, structural, and data modeling views, however, the key point is that software 
architecture design is the product of a multi-dimensioned process composed of independent 
and orthogonal views, impacted by variability, either planned or unintended. 

An architectural style (e.g., pattern) is a specialization of element and relation types, 
together with a set of constraints on their use. In this research effort, an architectural style 
consists of a set of constraints on the architecture, which define the set or family of archi¬ 
tectures, and satisfy them with a number of major styles. These styles may include general 
structures of pipes and fdters, layers, blackboards, object-orientation, and implicit invoca¬ 
tion, distributed systems; interactive systems; adaptable systems; and other styles including 
batch, interpreters, process control, specification-based, and rule-based. 

A sample of other recent advances in software architectures noted in Table 8-2 in¬ 
clude [Kru95] describing four views of software architecture for system building; the col¬ 
laborative work of [SNH95] who identified four additional industrial use views; four busi¬ 
ness views [HSOOc], the development of viewpoints [lEEOOb] to designate the means used 
to construct individual views; and the three categories of views identified as viewtypes 
[CBB+OS]. [DN02] suggest that over the past several years, software architecture emerged 
as the appropriate level for addressing software quality, and that recent systemic efforts of 
understanding how the design patterns and architectural styles contribute to that quality 
analysis effort. In their research [DN02] provide a concise and recommended survey on 
eight representative software analysis methods in different domains, compare and contrast 
similarities and differences through study, comparison, and classification, and offer guide¬ 
lines supporting the use of the most suitable method for an architecture assessment process. 

Emerging software tools including XML and ADLs support current software archi¬ 
tecture research and development efforts with the potential for employing common coarser- 
grained architectural elements (e.g., components and connectors) and interconnectivity 
schemes for architecture-based development, and featuring formal modeling notations, 
analysis, and development tools operating on architectural specifications. However, the 
research community currently lacks consensus on ADLs, the capabilities expected from an 
ADL toolset, and what aspects of a software architecture to model. 
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IX, ANALYSIS OF MODEL CREDIBILITY IN FIVE HMDS M&S 


A. INTRODUCTION 

Chapter IX provides an overview of the Ballistic Missile Defense System (BMDS), 
domain M&S, a concise description of the BMDS domain background^^^, the BMDS do¬ 
main M&S hierarchy, a BMDS System-Level M&S synopsis, BMDS M&S demographics, 
and a review of the BMDS model representations populating the five BMD System-Level 
simulations under study in this dissertation. The BMD domain overview section provides 
the Agency background and highlights the significant role Department organizational 
changes played in the current status of BMD System-Level M&S. Building on the organ¬ 
izational outline and M&S domain overview, the chapter continues with a review on the 
Agency’s implementation of the Department’s policies for establishing simulation credibil¬ 
ity supporting confidence in BMDS simulation results, and W&A background information 
for each of the BMD System-Level M&S. 

Summary level information of the other M&S in the domain M&S hierarchy pro¬ 
vide additional context for the analysis. A top-level review of the BMD System-Level 
M&S fidelity, and the foundations for radar sensor [Mac92, Cla93, EBOl] fidelity follow. 
The analysis identifies additional root causes for heterogeneous anomalies in the BMD 
System-Level M&S. The research methodology supported by the NFS software engineer¬ 
ing distance-learning model facilitates the timely study of Department primary source ma¬ 
terial for software-intensive legacy simulation systems. This case study also employs se¬ 
lected product line practice areas (e.g.. Organization Management, Technical Management, 
Software Engineering) [Coh02] as a tailored framework for the missile defense domain 
analysis. The scope of this research includes five large-scale. Missile Defense Agency leg¬ 
acy simulations, identified later in the chapter, and supporting appendices. 


The JTA includes specific functional domains, separate from the JTA Core standards and guidelines and which may be inappropriate 
for systems in other domains, only to ensure interoperability within the domain. These domains include subdomains containing JTA 
elements applicable to systems within that subdomain. The intention of the JTA is for the systems in the subdomains and domains to 
eventually adopt the JTA elements in the JTA Core standards [JTA02a]. 
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B, 


SELECTION CRITERIA OF A RELEVANT DOD DOMAIN 


The selected Department M&S domain , the Ballistic Missile Defense System 
(BMDS), met the following research criteria: 1) the BMDS domain is instantiated in the 
Department’s Joint Technical Architecture [JTA02a] specifically in the weapon system 
(e.g., missile defense sub-domain), M&S, and C4ISR (e.g., space reconnaissance sub¬ 
domain) domains identified by the bold areas in Figure 9-1; 3) the selected BMDS domain 
is also currently engaged with multiple international cooperative M&S programs; 4) the 
BMDS domain is part of the Department’s Transformation process [RumOl] with trans¬ 
formation criteria established by [Rum02c, Ald02e]; 5) the BMDS is a high-risk, mission- 
critical defense capability (Figure 9-2) for the Nation, allies, and friends; 2) the Missile De¬ 
fense Agency (MDA), responsible for the BMDS is a jointly organized Department domain 
including Army, Navy, and Air Force systems;6) the BMDS domain will implement the 
Department’s evolutionary acquisition strategy [Ald02d]; with the acquisition strategy cri¬ 
teria established by [Rum02c, Ald02e]; and 7) the BMDS domain possessed an established 
hierarchy of M&S, introduced in Figure 9-3, capable of supporting an analysis of the De¬ 
partment’s legacy architecture for reverse engineering, reengineering, and reuse (R3). 



Figure 9-1. JTA Domains Included in this Study (After [JTA02a]) 
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In this context a domain is an area of knowledge or activity characterized by a set of concepts and terminology understood by practi¬ 
tioners in that area. 

The acronym C2/BM represents the BMDS C4ISR function in this work. 
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C. ORGANIZATION MANAGEMENT 

1. Domain Overview of the Organization Structure 

In order to meet the ehanging ballistie missile threat to the Nation and adapt from a 
Cold War monolithic threat environment to the current Post-Cold War asymmetrical 
threats, the Nation has continued to evolve its missile defense capabilities. This evolution 
began with President Reagan’s Strategic Defense Initiative in 1983, which established the 
Strategic Defense Initiative Organization (SDIO), continued with the establishment of its 
successor, the Ballistic Missile Defense Organization (BMDO) [BMDOOa], and most re¬ 
cently, the current Missile Defense Agency (MDA) [Rum02c, Ald02e]. As the world geo¬ 
political situation changed in the 1980s and 1990s, national-level policy and defense strat¬ 
egy reordered the priorities and focus of the Agency several times. 

In addition, the complicated roles and responsibilities of a joint program integrating 
service-led major defense acquisition programs (MDAPs), a cumbersome Department re¬ 
quirements generation and acquisition process, international treaty constraints, and an in¬ 
consistent budget process affected the BMD system development and the supporting M&S 
effort. The current state of missile defense system representations in the BMD System- 
Level M&S, discussed in this chapter, occurred partly as the result of the many organiza¬ 
tional impacts and Department process constraints. 

Since the 1980’s the Agency’s models and simulation program reflected changing 
National missile defense priorities evolving from the bi-polar Cold War environment to 
counter the myriad of emerging and uncertain threats of the post-Cold War world; 

• 1984-1987 - Explore technologies for national ballistic missile defense, 

• 1987-1991 - Start acquisition of a phased national ballistic missile defense 
(NMD), 

• 1991-1993 - Acquire a limited global ballistic missile defense (GPALS), 

• 1993-1996 - Develop and field a Theater Missile Defense (TMD). Continue NMD 
as a technology readiness program, 

• 1996-2001 - Continue to acquire TMD. Develop a NMD system for possible lim¬ 
ited deployment [BMDOOa], 

• 2002-Present -Develop and field an integrated Ballistic Missile Defense System 
(BMDS) [Rum02c, Ald02e]. 
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At the turn of the century the Agency managed separate TMD and NMD system architec- 
tures , and lacked a single unified missile defense system architecture, while the Agency 
executed three concurrent, and often-competing acquisition strategies: 

• Field TMD systems quickly, 

• Determine the deployment strategy for the NMD systems, 

• Continue to advance the technology [BMDOOa], 

The Service Components TMD programs, and Joint Program office-managed NMD 
program developed their respective missile defense system efforts as MDAPs, with the 
BMDO responsible for integrating the diverse, complex weapon and sensor systems into 
single, seamless interoperable operational system architecture [BMDOOd]. The Department 
designed the Army’s PATRIOT program (e.g., PAC-2, PAC-2 GEM, and PAC-3) and 
Navy Area Defense (NAD) lower-tier missile defense systems to defend against terminal 
endoatmospheric threats. At higher altitudes, the Department devised the Army’s Thea¬ 
ter Area Air Defense System (THAAD) and Navy Theater Wide (NTW) missile defense 
systems to counter upper-tier exoatmospheric missile threats [BMDOOd]. 

The NAD and NTW missile defense systems evolved from the Aegis Weapon Sys¬ 
tem’s primary mission as a fleet air defense system [BMD99a]. Other missile defense ef- 
forts included the Airborne Laser (ABE) program, an Air Eorce managed system. The 
PAC-3, THAAD, and NTW systems employed hit-to-kill (HTK)^^^ technology. Appendix 
A summarizes the mission, organization, technical, acquisition, management, procedures 
processes supporting the Agency’s evolution and organizational responsibilities in the 
1996-2001 timeframe. 

The Agency also supported three major international collaborative programs includ¬ 
ing an international cooperative program with Germany and Italy, the Medium Extended 
Air Defense System (MEADS)^^*’. The Israeli ARROW program [BMDOOj], now deployed 
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System architecture in this context is the software architecture plus execution and development environments. 

The PAC-3 and NAD missile defense weapons employed against endoatmospheric threats are referred to as lower-tier systems. 
Endoatmospheric refers to being within the Earth’s atmosphere; generally considered tro be altitudes below 100 km [MDA02b]. 
THAAD and NTW are also called mid-course systems 

Exoatmospheric refers to being outside the Earth’s atmosphere; generally considered to be altitudes above 100 km [MDA02b}. 
The ABL concept employed a multimegawatt chemical laser to destroy missiles in the boost phase [BMDOOa]. 

Hit-to-kill technology (HTK) employs kinetic energy to destroy the target [BMDOOa]. 

MEADS was conceived to provide deployed NATO maneuver forces with 360 degree protection [BMDOOa]. 
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and operational is another international missile defense effort developed with U.S. support, 
and employs a high-explosive warhead rather than the HTK teehnology designed into U.S. 
missile systems. The Russian-Ameriean Observable Satellite (RAMOS) is a cooperative 
effort to observe the earth’s atmosphere and ballistic missile launches. 

However, as the United States enters the 2U' century, the dynamic geo-strategic en¬ 
vironment no longer checked by bi-polar world power control presents new challenges. 
New and uncertain 2U' century threats such as missile attacks from rogue nations, terror¬ 
ism, and weapons of mass destruction (WMD), brutally driven home to the American Na¬ 
tion on September 11, 2001, eclipse the Cold War focus on a massive Soviet missile attack. 

2, A MISSION-CRITICAL SYSTEM IN THE NATIONAL DEFENSE 

On January 2, 2002, the Secretary of Defense [Rum02c] established the Missile De- 
fense Agency (MDA), revised the agency concept of operations , and directed the initia¬ 
tion of a single joint program to develop an integrated ballistic missile defense system 
(BMDS). [Rum02c] also directed a capability-based requirements process, supported by 
the full and cooperative efforts of the Services, Joint Staff, and defense agencies to achieve 
the objectives of this National priority (Appendix B). 

The Department’s organizational change of the BMDO to the MDA was part of a 
transformation process to replace an overly restrictive, non-responsive, and overly prescrip¬ 
tive requirements and acquisition process [Ald02e]. These significant policy changes re¬ 
moved the communication barriers with the former MDAPs, and permitted a two-way dia¬ 
logue establishing a foundation for improved system representations in the BMDS System- 
Level M&S. However, the creation of the BMDS also added: an expanded mission, a 
wider scope of ballistic missile defense responsibility, many new complex program re¬ 
quirements, and an accelerated milestone schedule placing renewed emphasis on the devel¬ 
opment of a credible BMDS M&S program. 

Nearly one year later on December 17, 2002, President Bush identified the impor¬ 
tant role of missile defense to the Nation, friends, and allies, and directed the Secretary of 
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[Rum02c] described the new MDA organizational structure, roles, responsibilities, processes, and policies that detailed the way the 
MDA operates. 
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Defense to “proeeed with the fielding of an initial set of missile defense eapabilities” 

[Bus02f]. In addition the President made the following statement underseoring the mis- 

sion-eritical nature of missile defense: 

The deployment of missile defenses is an essential element in our broader 
efforts to transform our defense and deterrence policies and capabilities to 
meet the new threats we face. Defending the American people against these 
new threats is my highest priority as Commander-in-Chief, and the highest 
priority of my Administration [Bus02f]. 

[Rum02c] requires a single missile defense program to design, develop, and test the 
BMD system elements (e.g., the former MDAPS) of an integrated BMDS to defend the 
U.S., deployed forces, allies, and friends, with a BMDS that layers defenses to intercept 
missiles in all phases of their flight phases (boost, midcourse, and terminal) against all 
ranges of threats [MDA02a, MDA02c, MDA02b, MDA02e, MDA02f, MDA02h, MDA02j, 
MDA02m, MDA02x, MDA02y, MDAOSb]; support the fielding of element capabilities, 
such as PATRIOT Advanced Capability-3 (PAC-3) system, as soon as practicable; de¬ 
velop, test technologies and improve missile defense test ranges [Pat99, TRGOl]; and pro¬ 
vide early capability, if necessary, inserting new technologies as they become available, or 
when the threat warrants [Bus02e, Rum02c, Kad02b, Kad02c, MDA02i, MDA03e]. The 
Under Secretary of Defense (AT&L) [Ald03] noted, “space and missile defense is central 
to the future of our national security” [Ald03], reconfirming missile defense as a mission- 
critical requirement for the Nation. 

In addition to the HTK interceptor technology and directed energy weapons, the 
Agency is also developing sensor systems that will improve the ability to detect, track, and 
identify ballistic missile warheads from countermeasures, which will be integrated in the 
BMDS through a Battle Management / Command and Control (BM/C2) system. The 
Agency is also exploring advanced weapon capabilities including space-based lasers, and 
sea- and space-based kinetic energy systems [MDA02p]. 

Figure 9-2 illustrates the major BMDS functions required of an integrated collection 
of defense-in depth capabilities (e.g., detection, identification, classification, battle man¬ 
agement, sustainment, engagement, kill assessment) supporting the Boost Phase Defense 
Segment (BDS), Midcourse Defense Segment (MDS), and a Terminal Defense Segment 
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(TDS) (e.g., the missile defense kill-ehain). A planned Missile Defense Test Bed in the 
Paeific will support these three segments with additional test realism, the ability to support 
multiple engagements, and provide a limited contingeney capability. Appendix B provides 
a synopsis of U.S. missile defense acquisition programs, international collaborative efforts, 
technology programs, and the BM/C2, weapon, sensor, and communication components of 
these programs in 2002. 


Midcourse Phase 
Defense Segment 



Figure 9-2. BMDS Defense Phase Segments and Required Capabilities (After [RayOlc]) 

3. A HIGH-RISK, SOFTWARE-INTENSIVE SYSTEM 

Missile defense has other hurdles including the software challenge identified by 
Lieutenant General George Monahan for the SDI program in his 1990 report to the Secre¬ 
tary of Defense when he noted. 

The greatest engineering, vice technical, challenge in the SDI program is 
software... [BMDOOa]. 

BMDS software-intensive systems will be complex, expected to support system evolution, 
perform the most difficult tasks, including battle management, recover from software and 
hardware failures, and respond correctly to anticipated and unanticipated threats to the sys¬ 
tem. Critics and supporters agree that software errors will occur in a system as complex as 
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the BMD, but argue over whether the failures would be catastrophic^^^ [Bow02]. [ParOl] 
believes the development of dependable, trustworthy software-intensive systems for the 
BMDS is a very high-risk undertaking and may very well be impossible. 

Correctly assessing and meeting the many challenges for developing quality soft¬ 
ware-intensive systems supporting systems engineering/integration, BMD System testing, 
operations, and simulation development with credible verified and validated software prod¬ 
ucts is on the critical path to fielding an integrated missile defense system. In addition to 
supporting the evolutionary acquisition of the BMDS program and fielding of block capa¬ 
bilities every two years, the evolving BMD M&S program will also support all missile de¬ 
fense life cycle support requirements. 

D, BMD SYSTEM M&S DOMAIN OVERVIEW 

The Agency M&S program [MDA03M] supporting the development of the BMDS 
evolved from the large-scale, legacy M&S development efforts of previous missile defense 
programs [SCC+88, SW96]. The current M&S programs has three categories mapped to 
the Department’s M&S Hierarchy in Figure 2-2. The three categories of the Agency’s 
four-level M&S Hierarchy illustrated in Figure 9-3 are the: BMD System Threat, Signa¬ 
ture, Environment, and Lethality (TSEL). The BMD System TSEL M&S, at the bottom 
layer of the Agency’s four-level hierarchy support the development of the BMD Element- 
Level components/sub-systems/systems, at the second layer of the hierarchy. The BMD 
Element-level M&S directly support the Element programs. The BMD System-Level 
M&S at the third level of the hierarchy M&S supports the system-wide integration and in¬ 
teroperability of the element representations into the BMD system. 

The BMD System-Level M&S and the BMD System TSEL comprise the BMDS 
Core Model set. The BMD System-Level M&S is the only category addressed in detail 
within this dissertation, while the Element-Level M&S, and BMD System Threat, Signa¬ 
ture, Environment, and Lethality M&S are summarized at a very high level. See Chapter 


Several meanings of catastrophic have been developed, but [SCC+88] defined a catastrophic failure as a decline in system perform¬ 
ance to a 10 percent or less of expected performance. 
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VII for a summary of the Department’s Joint simulation development program, (e.g., 
JWARS JSIM, and JMASS), the Ageney’s capstone M&S category shown in Figure 9-3. 



Figure 9-3. BMD M&S Addressed in this Study (After [PMR97]) 

1. BMDS Legacy Model and Simulation Systems 

A major MDA objective is to provide credible M&S and improve the Department’s 
confidence in BMDS simulation results [Kad02a, Kad02b, Kad02c]. However, the BMDS 
M&S program has four major challenges to overcome in support of the vision to make mis¬ 
sile defense a reality: 1) developing credibility in the M&S and building user confidence in 
the results, 2) accurately modeling the significant technological hurdles inherent in missile 
defense, 3) dealing with major acquisition program complexity, and 4) the software engi¬ 
neering challenge itself 

The Agency and its predecessor Service / Agency / Component organizations de¬ 
veloped an expensive portfolio of M&S including large-scale, legacy simulation systems: 

Adage, Carmonette, and COMO III [RBB-l-82, Che87] to support the air and missile de- 
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fense mission. Many of these M&S evolved in an ad hoc manner, evolving into large sys¬ 
tems with multiple stakeholders and significant support infrastructures. A significant num¬ 
ber of the existing systems lacked formal documented conceptual models and reflected an 
inconsistent V&V history [RRB-l-82, Che87]. 

In addition, the number of M&S tools in the missile defense domain proliferated in 
the 1990’s until there were nearly three hundred M&S systems competing for BMDO M&S 
life cycle funding and support [BMD99b]. The Agency’s legacy M&S systems supported 
experiments, analysis, studies, and tests with little documented V&V, resulting in reduced 
credibility of the accreditation process for these simulations, and limiting confidence in ex¬ 
periment, analysis, study, or test results [RRB-l-82, Che87]. 

In a series of agency-level M&S reviews conducted duringl998 andl999, redun¬ 
dant, low-use M&S, and simulation systems with a low potential for future integration or 
HLA interoperability were discontinued, while approximately eighty-eight M&S tools were 
determined to be MDAP-unique and management responsibilities were assigned to the 

former major defense acquisition programs (e.g., PATRIOT, THAAD) [BMD99b]. The 
MDAP-unique tools currently constitute the MDA Element-Level M&S category of the 
MDA M&S Hierarchy in Figure 9-3. The remaining common-, and general-use legacy 
tools, supporting multiple internal and external MDA stakeholders, employed in all phases 
of the BMD system evolutionary program development (e.g., RDT&E, Transition, Opera¬ 
tions & Maintenance phases), and integrated into multiple functional areas (e.g., analysis, 
training, experimentation, acquisition, and operations), became the BMDS Core M&S. 
Figure 9-3 illustrates the two categories of the BMDS Core M&S: the BMDS Threat, Sig¬ 
natures, Environment, Lethality and Threat (TSEL) category and the BMDS System-Level 
M&S. 


2, BMDS System-Level Legacy M&S Systems 

The five simulations comprising the MDA System-Level M&S layer of the MDA 
M&S Hierarchy [SW96] shown in Figure 9-3 comprise the scope of this research. The sys¬ 
tems under study are: the Commanders Analysis and Planning Simulation (CAPS), the 
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MDAP-unique M&S were determined to be valid requirements for an individual MDAP, but did not have system-wide applicability. 
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Missile Defense Wargame and Analysis Repository (MDWAR), the Extended Air Defense 
Simulation (EADSIM), the Extended Air Defense Testbed (EADTB), and the Missile De¬ 
fense System Exerciser (MDSE). Theoretically, the BMD System-Eevel M&S in this layer 
include representations of the Element-Eevel systems at varying degrees of fidelity, accu¬ 
racy, precision, and resolution. In the future, the BMD System-Eevel M&S will in turn 
support development of valid BMDS representations in the Department’s capstone-level 
joint M&S. 

The BMD System-Eevel M&S support environments^"^® are repositories of missile 
defense domain expertise, critical intellectual property, and the foundation for the devel¬ 
opment of future components in the missile defense domain core asset portfolio. This con¬ 
stituency of knowledgeable stakeholder supporters is almost always lacking in the devel¬ 
opment of new systems, where requirement-stakeholders often compete for priority, fund¬ 
ing, and allocation of development resources. The MDA also supports external stake¬ 
holders including the Department’s operational test community. National-level decision¬ 
makers, and international cooperative partners. 

a. Commanders Analysis and Planning Simulation (CAPS) 

The CAPS model is a theater level, force-on-force, missile defense system 
analysis tool, and qualitatively considered low-fidelity. CAPS simulates active theater de¬ 
fense against ballistic missiles and air breathing threats and is employed as a planning, 
training and analysis tool with four views: footprint view, operating area view, defended 
area view, and scenario view. These capabilities illustrate defensive footprints, or areas on 
the ground that may be protected by missile defense units, possible operating regions for 
defensive systems, areas defended by various positioning of defensive forces, and expected 
performance of missile defense systems in a campaign. See Appendix C for additional in¬ 
formation. 
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The support environment includes the multiple stakeholders, both government and private sector, who manage, develop, test, Vc&V, 
use, or support the M&S development process. 
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b. Missile Defense Wargame and Analysis Resource (MDWAR) 

The MDWAR simulation, formerly called WARGAME 2000, is a consid¬ 
ered a low-medium fidelity simulation designed to provide the missile defense community 
with a Human-In-Control (HIC)^"^^ wargaming capability. MDWAR provides a construc¬ 
tive M&S tool to develop missile defense concepts of operation CONOPS), doctrine, tac¬ 
tics, techniques and procedures^"^^ (TTPs) through the use of virtual experimentation in a 
synthetic environment. MDWAR is the successor system to ARGUS simulation, devel¬ 
oped to provide a real-time, discrete-event simulation capability supporting the develop¬ 
ment of missile defense [TOM99]. MDWAR also supports missile defense architecture 
evaluation, joint missile defense exercises, and the development of the BMDS BM/C2 ele¬ 
ment. 

MDWAR HIC experiments interactively simulate the complex TMD and 
NMD threat environment with scenarios designed to induce the effects the real world con¬ 
fusion to the operators manning realistic operator consoles and displays. The objectives of 
the MDWAR war games are to exercise command and control (C2) processes in a HIC en¬ 
vironment with realistic battle management defense system features that enable the opera¬ 
tors to perceive and interpret the battlefield situation, identify problems, develop courses of 
action, and allows operators to select, implement and monitor the corrective action and its 
impact on the situation. See Appendix D for additional information. 

c. Extended Air Defense Simulation (EADSIM) 

EADSIM is a constructive simulation first released in 1989 [JAS97c] as a 
system-level simulation providing a many-on-many theater-level simulation of air, space, 
and missile warfare. EADSIM is generally considered a medium fidelity simulation. 
EADSIM models joint and combined force air and missile defense warfare, ranging from 
few-on-few to many-on-many theater scenarios, to determine the effectiveness of air and 
missile defenses against the full spectrum of threats as an analysis tool and to augment 
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Also referred to as human-in-the-loop (HIL) 

Computer-based TTP, CONOPS, and doctrine development supports the force and combat development functions of the services and 
joint forces and provides an early opportunity for the war fighter to influence the system development and employment. TTPs are also 
reviewed early in the material development process to see if a change to TTPs CONOPS, or doctrine negated the need for a materiel 
solution. 
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training. EADSIM supports the four pillars of theater missile defense: active defense, pas¬ 
sive, defense, attack operations, and BM/C2. EADSIM also models the command and con¬ 
trol (C2) decision processes, intelligence gathering, and the inter-platform communication 
process. See Appendix E for additional information. 

d. Extended Air Defense Testbed (EADTB) 

The EADTB simulation, an event-driven, Monte Carlo constructive simula¬ 
tion framework, supports missile defense modeling from the fire-unit to the theater level 
[BMDOl]. EADTB evaluates the effectiveness and efficiency of weapon systems against 
targets and the value of different force structures and resources. The EADTB system de- 
sign offers object-based simulation architecture with the flexibility to use high- or low- 
detail, user-developed, specific system representations (SSRs)^"^"^, for play on a simulated 
game board. [SMP98, Ray02a]. EADTB, considered medium-to-high fidelity is capable of 
supporting a wide range in the level of SSR detail in a single simulation exercise, running 
interactively or in batch mode, and provides analytical flexibility, with the objective of re¬ 
ducing the need for multiple simulations [BBB-l-96]. See Appendix E for additional infor¬ 
mation. 


e. Missile Defense System Exerciser (MDSE) 

MDSE, originally called the Theater Missile Defense System Exerciser 
(TMDSE) [Too94] addressed the objectives, architecture and engineering considerations of 
the theater level missile defense System-Eevel hardware-in-the-loop capability [BB98a, 
BB98b]. MDSE is the highest fidelity BMD System-level simulation. MDSE was origi¬ 
nally designed to demonstrate the interoperability of theater ballistic missile defense 
(TBMD) weapon systems and national sensor elements in a real-time, GPS-synchronized, 
centrally controlled, geographically distributed hardware-in-the-loop (HWIE) tool to test 
the TBMD family of systems (EoS) in a dynamic environment [Too94]. The original 
MDSE development concept specified by [Too94] identified the requirements for the first 
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EADTB is object-based, versus object oriented, since it does not support inheritance. 
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SSRs are data and code rule sets developed to represent objects (THAAD, PATRIOT) and control their behavior [BBB+96]. 
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three TMDSE builds and ealled for an ineremental build sehedule to develop a test resouree 
supporting interoperability and integration tests of the theater missile defense arehiteeture. 
The GMD (formerly NMD) arehiteeture was not a part of the original TMDSE require¬ 
ments, and will be addressed in future MDSE build requirements. See Appendix G for ad¬ 
ditional information. 

f. BMD System-Level M&S Status 

These simulations experieneed different growth patterns. In the ease of 
CAPS and EADSIM the doeumented start dates for simulation development do not refleet 
earlier development efforts. Eor instanee, CAPS evolved from the Operation Planning 
Simulation (OPS) [BBG+99f], while EADSIM built on the foundation of the earlier 
C3ISIM simulation [BBG+99e]. In eomparison to the CAPS and EADSIM evolutionary 
development, MDWAR and MDSE were more deliberate designs. The five BMD System- 
Eevel simulations under study are eurrently in the maintenanee phase of the M&S software 
life eyele and exeeuting to a stable program plan^"^^. 

E, BMD SYSTEM-LEVEL SIMULATION MODEL REPRESENTATIONS 

This seetion provides the analysis results illustrating the status of aeeeptable BMD 
System representations within the five BMDS System-Level M&S (e.g., CAPS, MDWAR, 
EADSIM, EADTB, and MDSE). The analysis ineludes the following missile defense rep¬ 
resentations: 

• Missile Defense Weapons - Terminal Defense Systems (TDS), Mideourse Defense 
Systems (MDS), and International, 

• Directed Energy Weapons - Airborne- and Space-based Boost Defense Systems 
(BDS), 

• Kinetic Energy Weapons - Sea- and Space-based Boost Defense Systems (BDS), 

• Radar Sensors - Terminal, Midcourse, and International systems, 

• Satellite / Infrared Sensors, 

• Airborne Laser Sensors, 

• Platform Systems 

• Geo-Spatial Environment representations. 


The current program plan for the M&S under study shows a steady state program of resources, factoring in inflation, for most of the 
decade. Any significant decrements to the projected program will extend the time needed to improve the credibility of the BMDS M&S. 
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For the purpose of this research an acceptable authoritative representation met the 
following criteria: the appropriate program office previously accredited the representation 
for a specific use, or the program office developed and documented the representation with 
an understanding of the representations’ planned intended use, or the program office re¬ 
viewed and accepted the system representation, with specific caveats for some documented 
intended use. Documentation supporting the PAC-3 Initial Operational Test and Evalua¬ 
tion (lOT&E) Interoperability Demonstration (PHD) [CRC02a, Bar02a, CRC02b, Bar02b] 
conducted in April 2002 proved invaluable for the purpose of assessing representation ac¬ 
ceptability. Based on this criteria, a “Y” in the following tables indicates evidence of at 
least one Acceptable System Representation, and an “N” indicates an Unacceptable / Miss¬ 
ing System Representation. These statistics do not state, imply or suggest a blanket ap¬ 
proval of system representation certification, accreditation, or validation by the respective 
MDA element program office for any future uses. 

1. BMD System-Level M&S TDS Weapon System Representations 

In the current TDS population of six missile systems identified in Table 9-1, five 
systems (83%) have acceptable representations of the specific missile in the BMD System- 
Eevel core simulations. One specific representation configuration of the THAAD EMD 
missile system is currently available at a very high level of aggregation in CAPS. Overall, 
eighty-seven percent (26/30) of the TDS missile representations are included in the BMD 
System-Eevel M&S. This is the highest percentage of weapon system representations 
found in the BMD System-Eevel M&S. Only the TDS sensor system representations, dis¬ 
cussed later in the chapter, exceed this metric in the System-Eevel models. The relatively 
high percentage of missile representations within the TDS may be attributed to the follow¬ 
ing: 

• A focus on terminal-phase systems (PATRIOT, THAAD, Aegis) during the devel¬ 
opment phase of the simulations, 

• An emphasis on interoperability among the different terminal-phase systems, 

• The requirement for a Single Integrated Air Picture (SIAP) including theater ballis¬ 
tic missiles, cruise missiles, and aircraft, both enemy and friendly. 
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• A missile defense program priority during most of the 1990s to defend U.S. forees 
and allies in a theater-wide operational seenario, 

• The PATRIOT and the Aegis terminal defense weapon systems were the most ma¬ 
ture missile defense systems, 

• A higher probability of M&S reuse within a mature program office with a family of 
related systems, such as the PATRIOT program office, 

• Support contractor longevity and domain expertise for different generations of simi¬ 
lar missile system (e.g., PAC-2, PAC-2 GEM, PAC-3) [BRC+01]. 

Of significance, a specific system representation based on correct specifications for 
the system at certain stage of the development cycle, even with an accurate verified and 
validated conceptual model may be completely invalid for another intended use. Table 9-1, 
illustrates this apparent inconsistency of specific system representations with the availabil¬ 
ity of the THAAD Program Definition and Risk Reduction (PDRR)^"^^ phase missile system 
version and the absence of the THAAD Engineering and Manufacturing (EMD ) phase 
missile system within the BMD System-Eevel simulation systems. 

Missing or inaccurate system characteristics or critical attributes [MDW+01] may 
adversely impact interoperability among the federation entities resulting in substantive in¬ 
teroperability issues including representational accuracy and internal sensitivity problems. 
There are several plausible reasons for the situation including a lack of coordination for a 
federation conceptual model, inadequate requirement management, poor configuration 
management, lack of system information, non-responsive contract support, funding short¬ 
falls, and scheduling. Whatever the root cause, the lack of specific valid system representa¬ 
tions may adversely impact the simulation or federations intended use and / or provide de¬ 
cision-makers with inaccurate simulation results. 


BMDS TDS MISSILE SYSTEM WEAPON REPRESENTATIONS 

TDS MISSILES 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

PAC-2 

Y 

Y 

Y 

Y 

Y 

PAC-2 GEM 

Y 

Y 

Y 

Y 

Y 

PAC-3 

Y 

Y 

Y 

Y 

Y 

THAAD PDRR 

Y 

Y 

Y 

Y 

Y 
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PDRR is the early Program Definition and Risk-Reduction Phase of a system’s development. 
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EMD systems have progressed beyond PDRR, with more mature engineering specifications supporting manufacturing of the system. 
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THAAD EMD 

Y 

N 

N 

N 

N 

AEGIS SM-2 BLKIVA“* 

Y 

Y 

Y 

Y 

Y 

# Avail /Total/ % Avail 

6 / 6 /100% 

5/6/83% 

5/6/83% 

5 / 6 / 83% 

5/6/83% 


[SpaOO, 

1CM99, 

lBAC+95, 

[BBG+96, 

[Too94, 


SpaOl, 
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TOM97, 

RayOla, 
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Citation from 
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Table 9-1. 


erminal Defense Segment Missile Representations 


The BMDS System-level M&S program also supports two signifioant international 
simulation development aetivities eited in Table 9-2, and an international eollaborative 
M&S program detailed in Tale 9-17. Systems sueh as the Israeli ARROW Weapon System 
are deployed operational systems with real-world interoperability requirements, while 
MEADS is a joint eooperative German, Italian, and U.S. missile defense development pro¬ 
gram. 


BMDS IINTERNATIONAL TDS MISSILE WEAPON SYSTEM REPRESENTATIONS 

INTL MISSILES 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

ARROW (ISRAEL) 

Y 

Y 

Y 

Y 

Y 

MEADS (GE, IT, US) 

N 

Y 

N 

N 

N 

1 / 2 / 50% 

2 / 2 /100% 

1 / 2 / 50% 

1 / 2 / 50% 

1 / 2 / 50% 



[SpaOO, 

[CM99, 

[BAC+95, 

lBBG+96, 

lToo94, 


SpaOl, 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation from 

BBG+99f| 
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Rayo2a, 

TEMOO] 



TRWOOb, 
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RayOlb] 
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Table 9-2. International Terminal Defense Segment Missile Representations 


2. BMDS System-Level M&S MDS Weapon System Representations 

The MDS missile system weapon representations listed in Table 9-3 are under¬ 
represented in the BMD System-Level M&S. There are only four weapon system 
representations of the GMD ground-based intereeptor (GBI) and one Navy Leap ALI 
missile system, based on parameter files, resident in the BMD System-Level M&S. CAPS, 
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All AEGIS SM-2, weapon systems in the Navy Area Defense (NAD) program in 2001 were counted as TDS representations. With 
the evolution of the Navy Theater Wide (NTW) program into the midcourse defense segment (MDS) the AEGIS SM-3 and LEAP ALI 
were counted in the MDS category for this study. 
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MDWAR, and EADSIM, three of the relatively aggregated M&S include the Aegis SM-3 
missile. However, acceptable SM-3 missile representations are not yet available in the 
EADTB constructive simulations or in the MDSE hardware in-the-loop tool. Reasons for 
the lack of MDS missile representations in the BMD System-Eevel M&S include the 
following; 

• National-level mission defense priorities shifted several times during the 1990s per¬ 
turbing the acquisition strategies, funding levels, and priority of the different sys¬ 
tems and their associated M&S programs, 

• The GMD (former NMD) program was developed as a single joint program with a 
specific mission versus the family of systems (EoS) approach employed for theater 
level missile defense systems requiring interoperability, 

• The GMD/NMD mission was to defend the Ei.S. against ICBM-class missile 
threats, as opposed to the EoS TAMD / TMD / TBMD mission against shorter range 
missile, aircraft, and cruise missile threats in a theater of operations, 

• The GMD/NMD architecture was not required or designed to be interoperable with 
the EoS TAMD / TMD / TBMD architecture, 

• With the exception of MDWAR, the GMD/NMD missile representations were not 
included in the development of the legacy BMD System-Level M&S, 

• The GMD/NMD program developed the BMD System-Level M&S equivalent in¬ 
corporating model representations of the former NMD component systems (e.g., 
UEWRS, DSP, BM/C2) in the GMD/NMD system architecture 

• Overall, Table 9-3 shows only forty percent (8/20) of the MDS missile system 
weapon representations are included in the BMD System-Level M&S, with no ac¬ 
ceptable MDS missile system representations available in EADTB or MDSE. 


BMDS MDS MISSILE SYSTEM WEAPON REPRESENTATIONS 

MDS MISSILES 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

GMD GBI I W/EKV 

Y 


N 

N 

N 

GMD GBI 11 W/EKV 

Y 

Y 

N 

N 

N 

AEGIS SM-3 BLK lA 

Y 

Y 

Y 

N 

N 

NAVY LEAP ALI 

Y 

N 

N 

N 

N 

# Avail /Total/ % Avail 

4 / 4 /100% 

3 / 4 / 75% 

1 / 4 / 25% 

0 / 4 / 0% 

0 / 4 / 0% 
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The GBI model is a combination of fly out tables and Keplerian propagation with the boost phase GBl modeled as a unitary or single 
stage interceptor defined by a two-dimension fly out table. Interceptor burnout conditions are extracted from the flyout tables and modi¬ 
fied to compensate the for earth’s influence. A point-in-space navigation method is employed because the two-dimensional fly out tables 
cannot represent trajectory shaping. Constraints include endgame maneuver and the interceptor seeker is not explicitly modeled, al¬ 
though a field-of-view is calculated. Intercept occurs at the time of closest proximity to the target with intercept outcomes modeled with 
two probabilistic draws: probability of hit (Ph) and probability of kill (Pk), and do not represent complex functions of intercept geometry, 
velocity, or closing angle [TOM99]. 
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Table 9-3. BMDS Midcourse Defense Segment Missile Representations 


Major missile defense weapon system platforms (e.g., Aegis) must also accommo¬ 
date multiple configurations of weapons, sensors, battle management, or communication 
systems during the same timeframe. This is an example of real world interoperability chal¬ 
lenges. It is also potential root cause for substantive interoperability issues in federations 
involving the real-world AEGIS SM-3 BLK IA and NAVY LEAP ALI systems and their 
respective system representations in CAPS, MDWAR, and EADSIM illustrated in Table 9- 
3. In the example highlighted in Table 9-3, there is the potential for inadvertently introduc¬ 
ing the incorrect or “almost good” enough missile weapon system representation into a fed¬ 
eration, with a potential to produce results similar to the Ariane 5 mishap. 

3, BMD System-Level M«&S BDS Weapon System Representations 

Supporting the Department’s transformation process, evolutionary acquisition strat¬ 
egy, and an overall reduction in product cycle time, the Department’s M&S programs must 
also transform and reduce simulation development cycle time, especially for new and pro¬ 
posed systems such as the BDS. Under the Block Development concept supporting the 
fielding of missile defense capability by blocks in two-year intervals (e.g.. Block 2004, 
Block 2006.. .Block 2016) there is a requirement to support the system engineering process 
with an M&S capability providing credible results across the temporal spectrum. This 
would suggest that as the Department system engineering process and M&S processes ma¬ 
ture together, the system engineer would allocate future M&S requirements, including fi¬ 
delity, resolution, precision, accuracy attributes [MDW+01] to future blocks and the simu¬ 
lation community would respond with quality products supporting credibility and confi¬ 
dence. 
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The next eategory of BMDS model representations is the BDS Direeted Energy 
Weapon system eategory. In the BMD system boost defense segment today there is a 
shortfall of BDS directed energy (Table 9-4) and kinetic energy (Table 9-5) weapon system 
representations, with the exception of the Airborne Laser (ABL) representations. The 
PLASTR and ISAAC models represent the ABL system in CAPS [BBG+99e], while the 
ABL representation in MDWAR is instantiated through a DIS gateway with the ABL op- 
erator-in-the loop simulation residing at the Theater Air Command and Control Simulation 
Lacility. Overall, only thirty-three percent of BDS directed energy system representations 
are currently available in the BMD System-Level M&S. 


BMDS BDS DIRECTED ENERGY WEAPON SYSTEM REPRESENTATIONS 

DIRECTED ENERGY 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

AIRBORNE LASER 

Y 

Y 

Y 

N 

N 

SPACE-BASE LASER 

N 

N 

N 

N 

N 

# Avail /Total/ % Avail 

1 / 2 / 50% 

1 / 2 / 50% 

1 / 2 / 50% 

0 / 2 / 0% 

0 / 2 / 0% 
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[CM99, 
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lToo94, 


SpaOl, 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation from 

BBG+99f| 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEMOO] 



TRWOOb, 

TBE02a, 

RayOlb] 




TRW02b] 

TBE02b| 




Table 9-4. Boost Defense Segment Laser Representations 


The kinetic energy systems envisioned for the future of missile defense and listed in 
Table 9-5 have no acceptable representations available, and are maybe the most challeng¬ 
ing for the unprecedented missile defense system development effort. Unlike the earlier 
TDS and MDS systems, which built on previous legacy missile defense systems, sub¬ 
systems, and domain bodies of knowledge, future system capability development faces 
many unknowns. Conversely, since the future may hold fewer constraints from current 
stakeholders and provide more trade space flexibility, the system engineer has the potential 
to evolve the system into areas with the most promise, assuming that the BMDS M&S de¬ 
velopment process is responsive, timely, and credible. 
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BMDS BDS KINETIC ENERGY WEAPON SYSTEM REPRESENTATIONS 

KINECTIC ENERGY 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

SEA-BASE KINETIC 

N 

N 

N 

N 

N 

SPACE-BASE KINETIC 

N 

N 

N 

N 

N 

# Avail /Total/ % Avail 

0 / 2 / 0% 

0 / 2 / 0% 

0 / 2 / 0% 

0 / 2 / 0% 

0 / 2 / 0% 
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Table 9-5. BMDS Boost Defense Segment Kinetic Energy Representations 


4, BMD System-Level M«&S Weapon System Representation Summary 

Currently, there are fourteen U.S. BMD weapon systems in four separate categories 
(e.g., six IDS missile systems, four MDS missile systems, two directed energy systems, 
and two kinetic energy systems) and two international weapon systems modeled across the 
five BMD System-Level M&S. Projecting future requirements for US-only weapon sys¬ 
tem representations in the BMDS weapon system category, and assuming the current set of 
weapon systems, there is a potential requirement for seventy weapon representations in 
every block build (e.g.. Block 2004, Block 2006.. .Block 2016). Assuming a requirement 
by the BMDS system engineering stakeholders for future block system representations, and 
the need to retain configuration-managed copies of past and present system representation 
configurations, the potential population for credible BMDS weapon system representations 
may soon number in the hundreds. This suggests the need for a methodology to manage 
model representation variability with a disciplined configuration management process. 

Possible future requirements for additional acceptable BMDS representations are 
daunting when a review of the Block 2004 representation inventory reflects only thirty-six 
acceptable TDS, MDS, and BDS weapon system representations, or fifty-one percent of 
acceptable system representations available. The international weapon system programs. 
Arrow and MEADS, have sixty percent (6 for 9) acceptable system representations. These 
statistics do not indicate any level of weapon representation certification or validation by 
the respective program office. 
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5. 


BMD System-Level M&S Sensor System Representations 


The BMDS terminal defense segment radar sensors (Table 9-6) have the highest 
pereentage of system representations in the System-Level M&S with a one hundred pereent 
fill rate. The rationale for this high rate is the same reason provided earlier in this section 
for the TDS weapon systems. However, as with the missile systems weapon category there 
is a significant decrease in system representations in the MDS (Table 9-8) and BDS (Table 
9-9) segments, with the largest void in three simulations, EADSIM, EADTB, and MDSE. 


BMDS TDS RADAR SENSOR SYSTEM REPRESENTATIONS 

TDS RADARS 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

PAC-2 AN/MPQ-53 

Y 

Y 

Y 

Y 

Y 

PAC-3 AN/MPQ-65 

Y 

Y 

Y 

Y 

Y 

TPS-59 (USMC) 

Y 

Y 

Y 

Y 

Y 

TPS-75 (USAF) 

Y 

Y 

Y 

Y 

Y 

AEGIS SPY-IB/D 

Y 

Y 

Y 

Y 

Y 

# Avail /Total/ % Avail 

6 / 6 /100% 

6 / 6 /100% 

6 / 6 /100% 

6 / 6 /100% 

6 / 6 /100% 


[SpaOO, SpaOl, 

[CM99, 

lBAC+95, 

[BBG+96, 

[Too94, 


BBG+99f| 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation from 
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Table 9-6. BMDS TDS Radar Sensor System Representations 


The availability of CAPS and MDWAR sensor representations for the International 
TDS radars in Table 9-7 reflect the relatively low fidelity of radar system representations in 
CAPS and MDWAR. The ARROW sensor representations in EADTB and MDSE support 
international interoperability exercises and testing. Although the current fill rate for Inter¬ 
national TDS radar representations is sixty percent overall, the international M&S program 
is expected to grow significantly requiring the rapid addition of system representations of 
all types. 


BMDS INTERNATIONAL TDS RADAR SENSOR SYSTEM REPRESENTATIONS 

INTL TDS RADARS 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

ARROW RADAR 

Y 

Y 

Y 

Y 

Y 

MEADS FIRE CNTRL 

Y 

Y 

N 

N 

N 
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MEADS SURVLNCE 

Y 

Y 

N 

N 

N 

# Avail /Total/ % Avail 

3/3/100% 

3 / 3 /100% 

1/3/33% 

1 / 3 / 33% 

1/3/33% 


[SpaOO, SpaOl, 

[CM99, 

lBAC+95, 

[BBG+96, 

[Too94, 
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Table 9-7. BMDS International TDS Sensor System Representations 


The next three BMDS sensor system representation tables represent radar, satellite / 
infrared, and ABL-speeifie sensor systems. Table 9-8 through Table 9-9 develop a similar 
pattern, high availability of representations at the lower-fidelity speetrum and few represen¬ 
tations available for the higher-fidelity simulations, EADTB and MDSE. Table 9-8, MDS 
radar sensor representations aehieve only a fifty pereent fill rate (15/30). 


BMDS MDS RADAR SENSOR SYSTEM REPRESENTATIONS 

MDS RADARS 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

THAAD PDRR 

Y 

Y 

Y 

Y 

Y 

THAAD EMD 

Y 

Y 

Y 

N 

N 

COBRA DANE 

Y 

Y 

N 

N 

N 

UEWR 

Y 

Y 

N 

N 

N 

SEA-BASE XBR 

Y 

N 

N 

N 

N 

GBR/XBR 

Y 

Y 

N 

N 

N 

# Avail /Total/ % Avail 

6 / 6 /100% 

5/6/83% 

2 / 6 / 33% 

1 / 6 /17% 

1 / 6 /17% 


ISpaOO, SpaOl, 

[CM99, 

lBAC+95, 

[BBG+96, 

[Too94, 
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TOM99, 

TOM97, 

RayOla, 
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Citation from 
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Table 9-8. BMDS MDS Radar Sensor System Representations 


Table 9-9, BMDS satellite / infrared sensors, aeeount for only forty five pereent (9/20) of 
the possible sensor representations. 
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BMDS BDS SATELLITE / IR SENSOR SYSTEM REPRESENTATIONS 

SATELLITES / IR 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

DSP SATELLITES 

Y 

Y 

Y 

Y 

Y 

SBIRS-HIGH 

Y 

Y^so 

N 

N 

N 

SSTS (SBIRS-LOW) 

Y 

Y 

N 

N 

N 

RAMOS (US/RF Program) 

N 

N 

N 

N 

N 

# Avail /Total/ % Avail 

3 / 4 / 75% 

3 / 4 / 75% 

1 / 4 / 25% 

1 / 4 / 25% 

1 / 4 / 25% 


ISpaOO, SpaOl, 

[CM99, 

lBAC+95, 

[BBG+96, 

[Too94, 


BBG+99f| 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation from 
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Table 9-9. BMDS BDS Satellite / Infrared (IR) Sensor System Representations 


Table 9-10 lists the Airborne Laser (ABL) sensor representations, whieh attain a sixty per- 
eent availability rate (9/15) of ABL representations, assisted by embedded model and fed¬ 
eration teehniques. 


BMDS ABL SENSOR SYSTEM REPRESENTATIONS 

ABL SENSOR 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

BEACON ILLUM. 

Y 

Y 

Y 

N 

N 

TRACKING ILLUM. 

Y 

Y 

Y 

N 

N 

LASER RANGER 

Y 

Y 

Y 

N 

N 

# Avail /Total/ % Avail 

3 / 3 /100% 

3 / 3 /100% 

3 / 3 /100% 

0 / 3 / 0% 

0 / 3 / 0% 


ISpaOO, SpaOl, 

[CM99, 

lBAC+95, 

[BBG+96, 

[Too94, 


BBG+99f| 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation from 
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Table 9-10. BMDS Airborne Laser Sensor System Representations 


6, BMD System-Level M«&S Sensor System Representations Summary 

There are eighteen BMDS sensor systems modeled aeross the five BMD System- 
Level simulations. Out of a possible ninety sensor system representations, fifty-eight are 
available, a sixty four pereent availability rate. Individual sensor eategory statistics range 
from a high of one hundred percent for the five sensor system representations in the TDS 


SBIRS-High and SSTS/SBIRS-Low are represented in MDWAR by a federation with the Missile Defense Space Tool (MDST) 
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radar sensor category to a low of forty-five percent in the satellite / infrared sensor cate¬ 
gory. The international sensor system category had representations for sixty percent of the 
simulations under study. 

7. BMD System-Level M&S Battle Management System Representations 

The BMDS battle management (BM) software is critical to the success of missile 
defense, a fact recognized from the earliest SDI studies including the Fletcher Report 
[FMA+84] and Eastport Study Group [CCL+85], and its success depends largely on M&S 
support. Command and control (C2) is the human element of the BM/C2, supported by the 
automated BM functions and communications network. The BMDS Battle Management 
system representation will address the system wide BMDS BM capabilities including the 
engineering needed to resolve protocol differences between TADIL-J and TADIL-K mes¬ 
sage formats employed by the TDS and MDS segments respectively, integrate the separate 
battle managers developed by existing MDA programs, and support BMDS system-level 
future battle manager development. 

BMDS BM systems cited in Table 9-11 represent high risks for the BMDS. The 

Fletcher Report [FMA+84] provided strong justification for reliable, safe, effective missile 

defense software and concluded; 

Further emphasis is needed on simulation as a means to assist 
the design of battle management systems and software. Spe¬ 
cific work is needed on algorithms related to critical battle 
management functions [FMA+84]. 

Currently, the development of the BMDS System-Level BM is in the embryonic 
stage and not yet adequately represented in any of the BMDS System-Level M&S. How¬ 
ever, BMD element-level legacy BM systems exist and many of their representations exist 
with the BMD System-levels simulations. The five TDS battle managers allocated to the 
five BMD System-level simulations noted in Table 9-11 have twenty-five acceptable model 
representations out of a possible twenty-five instantiations for a one hundred percent avail¬ 
ability rate. Acceptable GMD BM representations reside in both CAPS and MDWARS, 
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although missing BM representations in EADSIM, EADTB, and CAPS, ereate an overall 
forty pereent BM system availability rate. 

The two BDS battle managers, ABE and SBE, available in only two of ten possible 
BM representations, refleet a twenty percent availability rate. Table 9-11 lists the available 
representations and percent availability for each simulation. The entire BM representation 
population in Table 9-11 has an overall sixty-four percent (29/45) availability. 


BMDS BATTL EMANAGER SYSTEM REPRESENTATIONS 

C2BM 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

BMDS C2BM - System-Level 

N 

N 

N 

N 

N 

PAC-2 BMC2 - TDS 

Y 

Y 

Y 

Y 

Y 

PAC-3 BMC2-PDB 5-1- - TDS 

Y 

Y 

Y 

Y 

Y 

AEGIS BASELINE 5 - TDS 

Y 

Y 

Y 

Y 

Y 

JTAGS/ALERT - TDS 

Y 

Y 

Y 

Y 

Y 

THAAD BMC3 - TDS 

Y 

Y 

Y 

Y 

Y 

GMD BMC2 - MDS 

Y 

Y 

N 

N 

N 

ABLBMC41 - BDS 

Y 

Y 

N 

N 

N 

SBL BMC2 - BDS 

N 

N 

N 

N 

N 

# Avail /Total/ % Avail 

7 / 9 / 78% 

7 / 9 / 78% 

5 / 9 / 56% 

5 / 9 / 56% 

5 / 9 / 56% 


ISpaOO, SpaOl, 

[CM99, 

lBAC+95, 

[BBG+96, 

[Too94, 


BBG+99f| 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation from 
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BBG+99e, 
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TEMOO] 
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Table 9-11. BMDS Battle Manager System Representations 


The International BM system representations for the ARROW and MEADS representations 
in Table 9-12, achieved a slightly lower fifty percent fill rate, with ARROW BM repre¬ 
sented at a one hundred percent fill. 


BMDS INTERNATIONAL BM SYSTEM REPRESENTATIONS 

INTL C2BM 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

ARROW BMC2 

Y 

Y 

Y 

Y 

Y25I 

MEADS BMC2 

N 

N 

N 

N 

N 

# Avail /Total/ % Avail 

1 / 2 / 50% 

1 / 2 / 50% 

1 / 2 / 50% 

1 / 2 / 50% 

1 / 2 / 50% 


The ARROW-MDSE interoperability program achieves this capability. 
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8. BMD System-Level M&S Communication System Representations 

The BMD system-Level M&S communication network representations in Table 9- 
13 include the Joint Data Network (JDN), Joint Planning Network (JPN), and the Joint 
Composite Tracking Network (JCTN). The BMD System-Level M&S, except CAPS, ex¬ 
plicitly model the JDN, which forms the communication network for the TDS. The ab¬ 
sence of explicit communication representations in CAPS does not degrade the simulation, 
since it is not a communications planner. EADTB has the highest percent fill of communi¬ 
cation representations, confirming its primary role as a communications interoperability 
tool. Overall the three communication networks have five of fifteen possible representa¬ 
tions in the BMD System-Level M&S, an availability rate of thirty-three percent. 


BMDS COMMUNICATION SYSTEM REPRESENTATIONS 

COMMUNICATIONS 

CAPS""" 

MDWAR 

EADSIM 

EADTB 

MDSE 

JDN^“ 

N 

Y 

Y 

Y 

Y 

JPN""" 

N 

N 

N 

N 

N 

JCTN""" 

N 

N 

N 

Y 

N 

# Avail /Total/ % Avail 

0 / 3 / 0% 

I / 3 / 33% 

1/3/33% 

2 / 3 / 66% 

1/3/33% 


[SpaOO, SpaOI, 

[CM99, 

lBAC+95, 

[BBG+96, 

[Too94, 


BBG+99f| 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation from 
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Table 9-13. BMDS Communication System Representations 
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CAPS network are generic with no explicitly documented network models, although the JDN is the implicit model [BBG+99f]. 
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The Joint Data Network (JDN) is comprised of DoD Link 11, 16, and 22 message protocols sharing sensor data between all subscrib¬ 
ers in near-real-time. 
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The Joint Planning Network (JPN) supports the broadcast of TRAP (TDS) and TIBS traffic and non-real-time status and planning. 
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The Joint Composite Tracking Network (JCTN) provides for the distribution of best quality engagement data to all subscribers, in as 
close to real-time as possible, and supports the extension of the Navy’s Cooperative Engagement Capability (CEC). 
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9. 


BMD System-Level M&S Platform System Representations 


The BMDS System-Level Model and Simulation Platform Representations data in 
Table 9-14 shows a sixty-four pereent availability rate of launeher platforms, with the 
highest density of available representations in the TDS systems designed for theater mobil¬ 
ity (e.g., PAC-2, PAC-3, THAAD) and the Aegis Weapon System. The CAPS model, as 
noted with the eommunication systems, operates at a high level of aggregation and does not 
require detailed platform information. The ABL aireraft, a YAL-IA 747-400, requires the 
development of a speeifie traek configuration in all the System-Level M&S in support of 
its current doctrinal employment strategy. 


BMDS PLATFORM SYSTEM REPRESENTATIONS 

PLATFORMS 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

ABL YAL-IA 747-400 

N 

N 

N 

N 

N 

THAAD LAUNCHER 

N 

Y 

Y 

Y 

Y 

PAC-2 LAUNCHER 

N 

Y 

Y 

Y 

Y 

PAC-3 LAUNCHER 

N 

Y 

Y 

Y 

Y 

AEGIS WPN SYS 

N 

Y 

Y 

Y 

Y 

# Avail /Total/ % Avail 

0 / 5 / 0% 

4 / 5 / 80% 

4 / 5 / 80% 

4 / 5 / 80% 

4 / 5 / 80% 


[SpaOO, SpaOl, 

[CM99, 

lBAC+95, 

[BBG+96, 

[Too94, 


BBG+99f| 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation from 


TRWOOa, 

BBG+99e, 

Rayo2a, 

TEMOO] 



TRWOOb, 

TBE02a, 

RayOlbl 




TRW02b| 

TBE02b| 




Table 9-14. BMDS Platform System Representations 


10, BMD System-Level M&S Geodetic Coordinate Representations 

Geospatial information provides the capability to describe a location or position, 
normally exchanged through the use of coordinate locations. Common, well-known geo¬ 
spatial reference systems support interoperability with accurate coordinate locations and 
support the transformation of a coordinate system into multiple definitions of geo- and non¬ 
geo-referenced space. The coordinate systems include: 

• Inertial coordinates, 

• Earth-centered, Earth-Fixed coordinates, 

• Eocal coordinates [WilOO]. 
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The inertial eoordinate systems (e.g., non-rotating eoordinates) have an orientation 
fixed in spaee, with the Earth-Centered Inertial Model (ECI) coordinate frame commonly 
employed to define the motions of ballistic missiles or spacecraft [WilOO]. Spatial informa¬ 
tion consists of wide population of measurements and data types. Earth-centric spatial lo¬ 
cations include an Earth-centric view, indicating that spatial locations are geospatial. Spa¬ 
tial locations also support the identification of a fixed reference frame rotating with the 
earth, the geographic or geodetic reference. Sensors may use local coordinate systems to 
measure the positions of other bodies relative to the position of the sensor(s) [WilOO]. 
Conversely there is also a significant amount of spatial data (e.g., astronomical, orbital, 
geomagnetic and local observations) with reference frameworks fixed on the observer, so¬ 
lar, celestial or other positional standards. Geospatial information interoperability requires 
that: 

• Coordinate systems be defined such that coordinates describe location uniquely, 

• A data transfer mechanism exists to transfer data to other defined coordinate sys¬ 
tems [Bir97]. 

The approximate shape of the Earth, originally defined by early navigators as a 
spherical body supported the development of geodetic coordinate (GDC) systems. How¬ 
ever, more accurate geodetic coordinates requires the modeling of Earth as an oblate sphe¬ 
roid or ellipsoid-of-rotation [Bir97]. GDCs, composed of the ellipsoid and the datum, re¬ 
late Earth-centered, angular, geodetic latitude^^^ and longitude^^^ to an actual point, near or 
on the Earth’s surface. The ellipsoid is a good approximation of the geoid and used for 
many mapping applications. Many ellipsoids model the earth, and each ellipsoid may be 
defined in many ways in reference to the Earth. The datum defines the location of the el¬ 
lipsoid with respect to the Earth. 

Modem GDCs, derived from satellite data, provide measurements defining Earth- 
based ellipsoids with accurate global data. The World Geodetic System 1984 [WGS84] 


Latitude is the angle subtended with the ellipsoid’s equatorial plane by a perpendicular through the surface of the ellipsoid from a 
point, with positive latitude north of the equator and negative latitude south of the equator [Bir97] 

257 

Longitude is defined as the angle measured about the minor, or polar axis of the ellipsoid from a prime meridian to the meridian 
through a point, positive to the east of the prime meridian (e.g., Greenwich, England) or negative if west of the prime meridian [Bir97]. 

The geoid is a physical surface of equi-gravitational potential corresponding to mean sea-level, extending into landforms as an ap¬ 
proximation of mean sea-level, providing surface from which elevations may be directly measured [Bir97] 
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developed by the National Imagery and Mapping Ageney (formerly the Defense Mapping 
Ageney), defines the eurrent Department standard datum and geoid, and employs a eonsis- 
tent global set of 3-dimensional station eoordinates inferring the loeation of an origin, the 
orientation of an orthogonal set of Cartesian axes, and a seale. The [WGS84] standard, 
eurrent as of 1996, uses the refined eoordinates of the permanent Department’s Global Po¬ 
sitioning System (GPS) monitor stations as the operational WGS-84 referenee frame. 


BMDS GEODETIC COORDINATE REPRESENTATIONS 

GEODETIC COORDINATE 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

WGS-84 

N 

Y 

Y 

N 

Y 

Round, Rotating Earth 

Y 

N 

N 

Y 

N 

Non-rotating Earth 

Y 

N 

N 

N 

N 

Earth Centered Inertial (ECI) 

N 

Y 

N 

N 

N 

Locally Spherical Earth 

N 

N 

N 

N 

N 

# Avail /Total/ % Avail 

2 / 5 / 40% 

2 / 5 / 40% 

1 / 5 / 20% 

1 / 5 / 20% 

1 / 5 / 20% 


[SpaOO, SpaOI, 

[CM99, 

lBAC+95, 

[BBG+96, 

[Too94, 


BBG+99f| 

TOM99, 

TOM97, 

RayOIa, 

McQ97, 

Citation from 


TRWOOa, 

BBG+99e, 

Rayo2a, 

TEMOO] 



TRWOOb, 

TBE02a, 

RayOlb] 




TRW02b| 

TBE02b| 




Table 9-15. BMDS Geodetie Coordinate Representations 


In addition, a new global model of the Earth’s gravitational field, the Earth Gravita¬ 
tional Model 1996 (EGM-96), replaeed the outdated WGS-84 gravitational model 
[WGS84]. WGS-84 is the eurrent JTA geospatial data interehange standard and preseribed 
with the EGM-96 for the BMDS positioning, navigation, and timing standards [WBE+OO]. 
The EGM-96 supports the aeeurate assessment of the Earth’s gravitational funetion sup¬ 
porting the predietion of an objeet’s state veetor in time, a eritieal eomponent and a possi¬ 
ble souree of heterogeneous system representation anomalies in threat and BMDS eompo¬ 
nent trajeetories, sensor target traeking, geodetie information exehange or target traek data 
(e.g., substantive interoperability anomaly) if missing or modeled ineorreetly [WBE+OO]. 
Table 9-15 lists the status of geode tie eoordinate representations in the BMD System-level 
M&S. Other environmental models eurrently under assessment for standardization and in- 
elusion in the set of BMD System-level M&S inelude loeal weather models, earth atmos¬ 
phere models, sun position, lunar position, and star eatalog [Wil02, Wil03]. 
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11. BMD System-Level M«&S Summary of Representations 


The summary of acceptable authoritative representations in Table 9-16 of the five 
BMDS simulations under study provides the following statistics: 

• All five M&S systems included representations in the weapon, sensor, communica¬ 
tion, platform, and BM/C2 categories or product lines. 

• CAPS accounted for the highest percentage of authoritative representations in the 
missile and sensor product lines with eighty-two percent and ninety-five percent, 
respectively; and tied with MDWAR in the BM/C2 product line with seventy-eight 
percent. The ninety-five percent level for the CAPS sensor product line was the 
highest percentage of authoritative representations in a product line found in the 
study. Conversely, CAPS also had the only two product line categories, the com¬ 
munication and platform with no authoritative representations. The level of aggre¬ 
gation and abstraction in authoritative representations common to both CAPS and 
MDWAR contributed to the CAPS and MDWAR statistics. 

• The weapon product line authoritative representations ranged from a high of eighty- 
two percent to thirty-eight percent. The mode for weapon authoritative representa¬ 
tions was thirty-eight percent. 

• The sensor product line authoritative representations ranged from a high of ninety- 
five percent to forty-one percent. The mode for sensor authoritative representations 
was forty-one percent. 

• The BM/C2 product line authoritative representations ranged from a high of sev¬ 
enty-eight percent to fifty-six percent. The mode for BM/C2 authoritative represen¬ 
tations was fifty-six percent. 

• The communications product line authoritative representations ranged from a high 
of sixty-six percent to zero. The mode for communications authoritative represen¬ 
tations was thirty-three percent. 

• The platform product line authoritative representations ranged from a high of eighty 
percent to zero. The mode for platform authoritative representations was eighty 
percent, the highest mode statistic in the study. 

• The five-simulation total product line authoritative representations ranged from a 
high of seventy-five percent to forty-seven percent. The mode for the five simula¬ 
tion total authoritative representations was fort-seven percent. 

• The five-simulation population including all product line authoritative representa¬ 
tions equaled sixty percent. 


BMDS SYSTEM-LEVEL PRODUCT LINE REPRESENTATION SUMMARY 


SYSTEM REPRESENTATIONS 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

TDS Missiles 

6 / 6 /100% 

5 / 6 / 83% 

5 / 6 / 83% 

5 / 6 / 83% 

5/6/83% 

International TDS Missiles 

2121 100% 

1 / 2 / 50% 

1 / 2 / 50% 

1 / 2 / 50% 

1 / 2 / 50% 

MDS Missiles 

4 / 4 /100% 

3 / 4 / 75% 

1 / 4 / 25% 

0 / 4 / 0% 

0 / 4 / 0% 

Directed Energy Weapons 

1 / 2 / 50% 

1 / 2 / 50% 

1 / 2 / 50% 

0 / 2 / 0% 

0 / 2 / 0% 
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Kinetic Energy Weapons 

0 / 2 / 0% 

0 / 2 / 0% 

0 / 2 / 0% 

0 / 2 / 0% 

0 / 2 / 0% 

WEAPON SUB-TOTAL 

13 /16 / 82% 

10 /16 / 63% 

8 /16 / 50% 

6 /16 / 38% 

6 /16 / 38% 

TDS Radars 

6 / 6 /100% 

6 / 6 /100% 

6 / 6 /100% 

6 / 6 /100% 

6 / 6 /100% 

International TDS Radars 

3 / 3 /100% 

3 / 3 /100% 

1/3/33% 

1 / 3 / 33% 

1/3/33% 

MDS Radars 

6 / 6 /100% 

5/6/83% 

2/6/33% 

1 / 6 /17% 

1 / 6 /17% 

Satellite / IR Sensors 

3 / 4 / 75% 

3 / 4 / 75% 

1 / 4 / 25% 

1 / 4 / 25% 

1 / 4 / 25% 

ABE Sensors 

3 / 3 /100% 

3 / 3 /100% 

3 / 3 /100% 

0 / 3 / 0% 

0 / 3 / 0% 

SENSOR SUB-TOTAL 

21 / 22 / 95% 

20/22 /91% 

13 /22/ 59% 

9/22/41% 

9/22/41% 

BMDS C2BM 

7 / 9 / 78% 

7 / 9 / 78% 

5 / 9 / 56% 

5 / 9 / 56% 

5 / 9 / 56% 

International C2BM 

1 / 2 / 50% 

1 / 2 / 50% 

1 / 2 / 50% 

1 / 2 / 50% 

1 / 2 / 50% 

SENSOR SUB-TOTAL 

8 /11 / 73% 

8 /11 / 73% 

6 /11 / 55% 

6 /11 / 55% 

6 /11 / 55% 

COMM SYSTEM SUB-TOTAL 

0 / 3 / 0% 

1 / 3 / 33% 

1/3/33% 

2 / 3 / 66% 

1/3/33% 

PLATEORM SUB TOTAL 

0 / 5 / 0% 

4 / 5 / 80% 

4 / 5 / 80% 

4 / 5 / 80% 

4 / 5 / 80% 

SIMULATION TOTAL 

42 / 57 / 74% 

43 / 57 / 75% 

32 / 57 / 56% 

27 / 57 / 47% 

26 / 57 / 47% 

POPULATION TOTAL 

170 / 285 / 60% 


Table 9-16. BMD System-Level M&S Representation Summaries 


Although the five simulations equate to a statistically insufficient sample to relate to 
the Department’s total M&S population, the results correlate with the numerous studies, 
reports, and papers cited in earlier chapters, suggesting the Department population repre¬ 
sentations as a whole, may be more unpredictable than authoritative. However, this study 
covers one hundred percent of the system-level model representation population, with a 
significant sample size of 285 representations, which viewed in the Table 9-16, forms pat¬ 
terns. [Fow97] defines a pattern as “an idea that has been useful in one practical and will 
probably be useful in others” [Fow97], and further suggests that the establishment of a 
common framework supports the concept of component reuse for information systems. 

Table 9-16 identifies potential patterns for several software architecture patterns in¬ 
cluding product families (e.g. weapons), software product lines (e.g. missiles, lasers), soft¬ 
ware product line systems (e.g. PAC-3, THAAD missiles), and a software product line ar¬ 
chitecture. The sixty percent availability rate of authoritative representations (e.g., model 
product lines) resulted from the causal reasons cited in earlier chapters and for the specific 
reasons noted in this chapter. This suggests that current Department simulation software 
development techniques, procedures, and practices are inadequate or incomplete, inconsis¬ 
tently implemented, or a combination of both factors. 
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The BMDS poses even more challenges to the current status. The BMDS program 
requires authoritative representations for simulations supporting the engineering of future 
BMDS block capabilities. The BMDS system engineering process requires authoritative 
representations for Blocks 2004, 2006, 2008, 2010, 2012, 2014 and beyond to explore and 
experiment with capabilities, and provide a means to extend the performance window of 
existing system. Due to live-test limitations, the BMDS System-Level M&S will provide a 
significant percentage of the future data needed to assess BMD System-wide interoperabil¬ 
ity and performance. The urgency of improving the BMDS System-Level M&S and add¬ 
ing more credible authoritative representations increased dramatically with the President’s 
decision to deploy a limited Missile Defense capability in 2004. At that time the BMDS 
System-Level M&S will support an operational Missile Defense for the Nation, allies, and 
friends. Chapter X provides a software architecture-based alternative to improve the level 
of authoritative representations in future BMDS System-Level M&S. 

F. BMDS INTERNATIONAL COOPERATIVE M&S PROGRAM 

The Agency’s international cooperative M&S program requires the development of 
system representations with the necessary detail, attributes [MDW+01], and goodness-to-fit 
must pass several hurdles from the formal approval of the international agreement permit¬ 
ting the initiative, completion of required disclosure statement, implementing the necessary 
contracting vehicles supporting the effort, and executing the actual model development ef¬ 
fort. The international cooperative M&S development effort must also address information 
assurance, language, cultural, and national priority issues, in addition to the normal cost, 
performance, and schedule project management concerns. All or any one of these condi¬ 
tions may provide the underlying conditions for an invalid federate or the creation of intol¬ 
erable representational anomalies, resulting in substantive interoperability issues. 

Table 9-17 illustrates the scope of international cooperative M&S initiatives sup¬ 
ported by the BMD System-Level M&S. This international M&S effort is a key compo¬ 
nent of the missile defense interoperability strategy with friends, allies, and supports evolv¬ 
ing international cooperation efforts with new partner nation initiatives such as the Russian 
Federation Model and Simulation Program (US/RF). Table 9-17 reflects the System-Level 

249 



M&S supporting the initiatives, and the type of international agreements authorizing the 
program. EADSIM supports every international eooperative M&S program, exeept the 
embryonie US/RF initiative, while NATO employs EADTB in missile defense interopera¬ 
bility studies. The ARROW-MDSE program directly supports the international terminal 
defense segment interoperability testing of the El.S. PATRIOT missile defense system with 
the Israeli ARROW missile defense system. 


COUNTRY 

SYSTEM-LEVEL M&S 

LEGAL BASIS 

CAPS 

EADSIM 

EADTB 

MDSE 

MOA 

MOU 

D 

F 

w 

s 

Australia 


X 







X 


Canada 


X 



X 






Germany 


X 

X 


X 


X 




Greece 


X 






X 



Israel 


X 


X 

X 






Japan 


X 





X 




S. Korea 


X 






X 



NAMEADSMA 


X 

X 



X 





NCSA 


X 

X 


X 






UAE 


X 






X 



UK 


X 

X 






X 

X 

Singapore 


X 






X 



Spain 


X 






X 



Netherlands 


X 






X 


X 

Turkey 

X 

X 



X 






NATO ES 

X 

X 

X 


X 






US/RE 

New Model and Simulation Development Program - Legal Agreement Pending 

TOTAL 

2 

16 

5 

1 

6 

1 

2 

6 

1 

2 


Table 9-17. BMD International Cooperative M&S Program 

G. BMDS ELEMENT-LEVEL MODELS & SIMULATIONS PROGRAM 


The major Element-Level M&S, listed in Appendix H directly support the program 
requirements of MDA element program managers. They are also components of the joint 
BMD System M&S Hierarchy, developed in accordance with the approved element Model 
and Simulation Support Plan (MSSP). Element-Level M&S employ BMD System-TSEL 
M&S in the development of their respective system, and support the development of accu¬ 
rate element representations in the BMD System-Level M&S [Kad02a]. 
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H. BMDS-LEVEL THREAT, ENVIRONMENT, SIGNATURE & LETHALITY 

MODELS & SIMULATIONS 


The Legacy BMD System Threat, Environment, Signature, and Lethality (TSEL) 
M&S are the foundation codes of the missile defense domain. These common-and general- 
use codes represent the highest fidelity simulations in the missile defense M&S domain and 
directly support the design, development, and testing of all missile defense element sys¬ 
tems. These codes require the highest level of credibility. In addition, the expanded battle 
space of the new BMD system adds significant new requirements, beyond the current ca¬ 
pability of the tools, necessitating major upgrades and new development programs. Ap¬ 
pendix I has a short list of the BMDS Threat, Environment, Signature, and Lethality M&S 
program. 

I. BMDS SYSTEM-LEVEL M&S W&A PROGRAM 

1. AGENCY M&S W&A EFFORTS (1986-2000) 

The BMD Core M&S program inherited a legacy of challenges, issues, and prob¬ 
lems requiring timely solutions to allow the successful evolution of credible Agency M&S 
supporting confidence in the simulation results. The agency supported multiple M&S re¬ 
ports, studies, analyses, and reviews to establish the credibility of the M&S supporting the 
unprecedented technical challenge of the missile defense domain including : [FMA+84, 
CCL+85, Bre88, SCC+98, HSW+92, EHH97, KBM+97, Li97, SEL97, SEH+97, EHH+98, 
WAA+98, ZHF98, BBG+99a, BBG+99b, BBG+99c, KGB+99, Li99, ISE99, WAA+99, 
RTM+99a, RTM+99b, RTM+99c, RTM+99d, RTM+99e, RTM+99f, RTM+99g, BDHOO, 
LEH+00, SBH+01, WelOO, BRC+01, CoyOl, EPOl, LWC+01, LHC+02, Mil02a, 

STG+02]. 

The Agency’s internally- and externally-generated reviews for existing simulation 
systems identified the same type of problems noted in the preceding generation of air de¬ 
fense simulations (e.g.. Adage, Carmonette, COMO III [Che87]). The results also con¬ 
firmed the lack of silver-bullet solutions, and closely paralleled the credibility issues and 
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systemic, persistent Department M&S problems outlined in the preceding chapters includ¬ 
ing; 

• Lack of Technical and Substantive interoperability 

• Need to improve V&V process, 

• Lack of documentation, including conceptual models, 

• Poor or nonexistent configuration management of models and data, 

• Inadequate understanding by users and decision-makers of model assumptions, 

• Limitations, and capabilities, 

• Insufficient model development practices, 

• Need for improved procedures to update and maintain models, 

• Need to improve how the M&S credibility is established, 

• Need to improve how the validity of the model is determined. 

Simulations need hard data from test programs. 

The study effort for the ballistic missile defense M&S research phase started in Sep¬ 
tember 2000 and included a review of the agency documents including multiple Agency 
M&S briefings, reviews, reports, staffing actions, test results, conversations, and assess¬ 
ments, cited above. The Agency’s VV&A issues during the 1990s mirrored the Depart¬ 
mental M&S VV&A issues. Documented V&V of the BMDO core model set during the 
1986-1999 timeframe was inconsistent and rare. Two specific studies representative of 
other Agency M&S studies spanning most of the 1990s include [HSW+92] and 
[BBG+99a]. 

In 1992 [HSW+92] reported on the SDIO’s Brilliant Pebbles‘ program effective¬ 
ness and the extensive use of simulations in support of system design, manufacturing and 
deployment; concluding the simulations were still immature, and used unproven assump¬ 
tions, and noting: 

The accuracy of Brilliant Pebbles’ simulations has not been 
validated by test results or verified by formal reviews of the 
simulations themselves [HSW+92]. 

The BMDO and Joint Theater Air and Missile Defense Organization (JTAMDO) 

conducted a joint a, M&S review in 1999 [BBG+99a] of selected BMDO core models. The 

report identified model validation among the key risk areas and noted: 

There is no BMDO-wide process for assuring adequate 
model validity. The degree of V&V performed is currently 
up to the organization funding the developer, and no stan- 
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dards are followed. There is no organized link with the TE 
[test and evaluation] community for obtaining data for vali¬ 
dation or to show that the models can/cannot predict or ex¬ 
tend testing. There has only been a minimal effort to com¬ 
pare model results on the same scenario (anchoring) and 
inadequate checkout of less detailed models against results 
from the more detailed models [BBG+99a]. 

The majority of the [BBG+99a] issues included systemic BMDO organizational, process, 

and technical issues. Accordingly, the joint BMDO/JTAMDO M&S review [BBG+99a] 

recommended an on going V&V process citing: 

V&V is another important activity not actively pursued at 
this time in a centralized manner, resulting in uncertainties 
about the robustness of our simulations and non-uniformity 
in the degree of V&V performed. There should be a 
planned central effort to validate our models, collect valida¬ 
tion data, and assess the developers’ processes in order to 
reduce program risk. A part of this program should be to 
link T&E activities in an active effort to obtain data for 
model validation and for making sure the simulations can 
validly be used to extend test data for model validation and 
for making sure the simulations can validly be used to ex¬ 
tend test data. Anchoring simulations results is another 
critical activity, which would improve our understanding of 
the models and of their differences and which would in¬ 
crease our confidence level or motivate corrections 
[BBG+99a]. 

[MS02] and [Lev02] address the two key issues of credibility and confidence in 

missile defense simulations, with [Lev02] noting: 

Weapon system confidence is being able to predict the sys¬ 
tem’s performance to within a quantified uncertainty or 
confidence interval [Lev02]. 

However, the current suite of BMD System-Level M&S experienced various inconsistent 
VV&A approaches as the BMD System M&S V&V Program^^^ matured [HSW+92, 
BBG+99a]. 

In 1999 [BBG+99F] identified that CAPS lacked a formal V&V process since first 


The Agency implemented a centrally managed verification and validation program of the core M&S in FY2002 establishing a V&V 
process to improve systemic credibility issues and improve user confidence in the Agency core M&S results. 
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released in 1994, and noted, “there is no indieation of the validity of the CAPS engagement 
logie to represent BMDO systems” [BBG+99f]. A doeumented CMMS/ FDMS for CAPS 
was never developed. However, CAPS was formally aecredited for specific use on two 
occasions: 

• 1994-1996 for the Theater Missile Defense Cost and Operational Effectiveness 
Analysis study, accredited by both the Army and the Navy, 

• 1997 - Accreditation as the interim Joint Defensive Planner (JDP) by the BMDO 
[BBG+99a]. 

MDWAR, the newest addition to the core model suite initiated a V&V program 
early in the development cycle. The first major MDWAR wargame, C2SIM99, met joint- 
accreditation requirements by the NMD program office and USSPACECOM for execution 
in November 1999. Eollow-on wargame efforts included the Theater Ballistic Missile De¬ 
fense Operator-In-The-Eoop Phase I war game exercise in July 2000, and Phase II war 
game exercise in September 2000, with both events meeting joint-accreditation require¬ 
ments from the NMD program office and USSPACECOM. The developer first published 
conceptual model documentation for MDWAR components in 1999 [TRW02a, TRW02b]. 

EADSIM developed a wide missile defense community acceptance throughout the 
1990s, although [JAS97c] noted in a comprehensive review that little formal V&V was 
complete, no documented V&V results, and verification limited by the lack of a detailed 
design specification against which to check the software code. [JAS97c] acknowledged 
that some V&V work may be in progress at the time or classified, although AEOTEC re¬ 
ported similar findings in 1994. EADTB credibility according to [JAS97c] depended al¬ 
most completely on face validation assessments and community acceptance. 

The first major study employment of the EADTB simulation was the BMDO Cost 
and Operational Effectiveness Analysis (COEA) study. A face assessment [BBB+96] re¬ 
view of EADTB for the COEA study was completed October 25, 1996 and identified 96 
issues and 34 product quality issues. [BBB+96] detailed major problems including: 

• The lack of requirements against which to measure the capabilities, 

• Excessive run-time, 

• A high learning curve, 

• Insufficient testing through November 1995, 

• Verification was limited to algorithms, code testing, and performance, with no at- 

254 



tempt at design verifieation. 

• An ineomplete draft W&A Plan, dated Mareh 14, 1994 had not been plaeed on 
eontraet. 

• V&V eontaetor had one person assigned. 

• No formal coneeptual model deliverable. 

• Doeumentation was a high eoneem with key doeuments outdated or missing 
[BBB+96]. 

[BBB+96] also reviewed EADTB V&V proeedures in the areas of ballistie missile 
flights and radar sensors, and noted the existing EADTB V&V method eompared results 
with other models sueh as EADSIM or COMO. [BBB+96] reeommended EADTB for an 
application “wring out” but not for the more critical and contentious missile defense cam¬ 
paign-level studies. 

The EADTB program established a formal V&V effort for the simulation’s CMS 
components in 1993 and initiated a follow-on V&V program for the system specific repre¬ 
sentations (SSR) in 1999. The reasons for the six year difference between the 1993 initia¬ 
tion of the EADTB CMS V&V program and the 1999 EADTB system representations 
V&V program highlight systemic issues identified in this study limiting the credibility of 
all Agency System-Eevel M&S during the 1990s: 

• In the competition for resources between developing new system requirements with 
supporting stakeholders, or support for the potentially high recurring costs of estab¬ 
lishing and maintaining a consistent V&V program, the delivery of new capabilities 
generated user-support, whereas, V&V program efforts provided no new capabili¬ 
ties or satisfied customers, and could be contentious, 

• The inherent complexity of V&V confused most stakeholders and decision-makers, 
detracting from V&V proponent’s ability to compete with other Agency require¬ 
ments for resources, 

• A concern by the MDAP program offices that their system could be misrepresented 
in a simulation, or the representation used for a purposes it was not designed or in¬ 
tended, such as system performance, 

• The time and resources supporting SSR development in the MDAP system engi¬ 
neering correspondingly reduced the MDAP’s resources for it’s primary mission, 
with little perceived return-on-investment, 

• The simulation developers had limited access to the MDAP program offices to re¬ 
search, design and develop credible system representations for Agency-level stud¬ 
ies, analyses, or tests. 
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• Discrete CMMS/FDMS, prescribed by [DoD94, DoD96] were not developed for 
any of the five simulations during the early life-eyele phases, ereating at a minimum 
the possibility of logieal inconsistency in the VV&A proeess, 

• Any efforts to determine the eredibility of a simulation or federation for intended 
use was necessarily ad hoc, expensive, and short-lived without a verified and vali¬ 
dated coneeptual model of the mission spaee upon whieh to base the verification 
and validation process for the FOM or software implementation, 

• An Agency trend developed in the 1990s to accredit a simulation without underly¬ 
ing coneeptual models of the mission spaee or V&V plans, 

• The Department’s accreditation proeess supported multiple accreditation options 
and it was aeeeptable to aeeredit the simulation by simply using it, 

• Faee validation or a review by subjeet matter experts of the simulation for an in¬ 
tended use beeame the de facto standard for establishing credibility and supporting 
user eonfidenee, 

• The V&V of eertain SSRs (e.g., threats, systems under development) was extremely 
complicated and required close eoordination with many ageneies, 

• Contraetors had no incentive to V&V simulations, and contractual language sup¬ 
porting V&V as a deliverable were few, 

• The previously discussed Year 2000 remediation effort and HLA compliancy work 
pushed V&V efforts further down in the requirement queue. 

2. AGENCY M&S W&A EFFORTS (2001-2003) 

Two BMD System-level M&S, EADTB and MDSE, supported a major-program 
testing event, the PAC-3 independent operational test and evaluation (lOT&E) with the 
PATRIOT lOT&E Interoperability Demonstration (PHD) in April 2002. The event was 
signifieant in that the Army Test and Evaluation Command aeeredited the Missile Defense 
System Exereiser [Bar02a, CRC02a], in a virtual hardware-in-the-loop stimulation tool 
role, and the Extended Air Defense Test Bed [Bar02b, CRC02b, KS02], as a eonstruetive 
simulation for a full-up five firing unit air defense battalion in a representative threat envi¬ 
ronment supplementing a Department lOT&E for the PAC-3 missile defense system. The 
PHD built on a highly suecessful PAC-3 Developmental test and M&S program [BRC+01]. 

The year-long preparation effort in support of the PAC-3 PHD required the full eo- 
operation of the Ageney M&S organizations, program offiees, simulation developers, 
VV&A agents, the operational test eommunity and many other stakeholders to develop, 
verify, validate, aeeredit or eertify the system representations as aeeurate for the intended 
use of testing PAC-3 interoperability. The interoperability assessment addressed PAC-3 
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intra-battalion interoperability, interoperability between the Patriot system and other ele¬ 
ments of the Army Air and Missile Defense task Force, and interoperability between Pa¬ 
triot and other Joint Service elements in a Joint environment scenario [Bar02a, Bar02b]. 

The EADTB and MDSE simulated Theater Missile Defense architecture with tactically 
representative threats, message traffic, and communications loading. The PHD met all test 
objectives and successfully completed all test runs-for-record ahead of schedule in April 
2002 . 

Based on past performance and it’s credibility in the missile defense domain, 

MDSE was incorporated into the Department’s evolving Joint Distributed Engineering 
Plant (JDEP) program [DahOla, DahOlb, DTOla, DTOlb, Eew02b, DC02, Rad02] support¬ 
ing improved interoperability of Joint Eorces, an example of large-scale reuse by the De¬ 
partment and a potentially significant return-on-investment. Even more importantly, the 
collaborative multi-stakeholder effort supporting the PHD set the foundation for systemi- 
cally developing credible system representations in all BMD System-level simulations with 
element program participation [Kad02a]. 

The Extended Air Defense Simulation supported the Joint Project Optic Windmill 
(JPOW) VII Exercise [Obe02] in September 2002, after completing a collaborative VV&A 
process with the full cooperation of the program offices. This exercise also marked the 
first employment of the BMD M&S Integrated Product Development Team (IPDT) concept 
with the objective of developing, verifying, and validating accurate, authoritative represen¬ 
tations of element capabilities in the System-Level M&S [Kad02a]. The BMD System- 
Level M&S V&V program incorporated the lessons-learned from [HSW+92, BBG+99a, 
Bar02a, CRC02a, Bar02b, CRC02b, Obe02] and several other reports cited earlier, and de¬ 
veloped centrally-managed W&A processes for system-wide events [Kad02a]. 

J. BMDS SYSTEM M&S FIDELITY 

1. Agency M&S Fidelity Background 1986 - 2000 

The Agency adopted a “family of systems” (EoS) approach to describe mutual sup¬ 
port over a hierarchy ranging from high resolution, limited scope models (e.g., physics 


level codes) at the base, through higher level overlapping levels of models with lower reso- 
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lution [SW96, BBB+96], similar to the Defense Systems Management Collage model 
[PMR94]. [SW96] proposed an analytical taxonomy supporting the FoS approach with six 
qualitative levels: system, encounter, engagement, mission, campaign, and global. 
[BBB+96] took a different approach, basically developing a two-dimension array with four 
major application categories: analysis, test and evaluation, exercises and wargaming, and 
engineering assessed at four qualitative levels of analysis: campaign^^®, force-on-force en- 
gagement , few-on-few engagement , and special domain performance. 

These qualitative approaches were not sufficient for the high-risk, mission-critical 
missile defense domain. A later 1999 MITRE Corporation-led review of the BMDO M&S 
program [RTM+99b] addressed two major BMDO M&S issues central to this dissertation. 
[RTM99b] recommended the Agency: 

• Identify the methodology for representing systems and functions (e.g., emplace¬ 
ment, search and track, engage, missile launch / fly out, terminal guidance, intercept 
and post-intercept assessment) critical to missile defense at each level in the M&S 
hierarchy, 

• Analyze the consistency of M&S representations of functions up and down the 
M&S hierarchy [RTM+99b]. 

Another related internal Agency report, [BBG+99b] noted the quality of each level 
of simulation depended upon the quality of the input from the lower level simulation or 
code and recommended: 

• An evaluation of the required levels of fidelity between program office high fidelity 
simulations and factor-driven campaign-level simulation tools, 

• The Agency develop an agreed upon range of fidelity for the kill-chain components 
(e.g., sensors, weapon algorithms, platforms, C4I) instead of a set fidelity level to 
close the “fidelity gap” between different organizations [BBG+99b]. 


Campaign: Characteristics include aggregated resolution, theater-level scope, many-on-many scenarios, run time speed many times 
faster than real-time, generally lower fidelity, simulating time span from days to months, includes all architecture components plus 
ground, air, and naval forces. Addresses logistics factors: resupply, reload, damage and repair, and weapon inventory issues. Examines 
system performance impact on the course, duration and outcome of the campaign [BBB+96]. 

Force-on-Force: Characteristics include minimal aggregation, theater level scope, many-on-many scenarios, medium-to-high fidelity 
(e.g., treatment may be mixed with three of freedom (3 DoF) modeled for some aspects while way points may be used for other trajecto¬ 
ries), and probably does not play intercept or end game performance. Simulation time span is from hours to a few days, running faster 
than real time to support trade studies, and usually provides data for campaign models [BBB+96]. 

Few-on-Few: Characteristics one-on-one engagements, no aggregation, limited scope, high system detail and fidelity of representa¬ 
tion for system under examination. Intercept end game played, simulation time form a span of minutes to a few hours, run time faster 
than real time to facilitate sensitivity analysis, examines system and subsystem performance. Provides data for Force-on-Force models 
[BBB+96]. 

Special Domain: Configured to examine particular phenomena, such as command and control, battle management, communications, 
with aggregation, scope, fidelity of representation, and run time requirements dependent on specific study or experiment objectives 
[BBB+96]. 


258 



2, An Evolving M&S Fidelity Framework 2000 - 2003 

The Agency developed a framework providing decision-makers with a background 
on the BMD System-level model capabilities, flexibility, resolution, accuracy, and preci¬ 
sion (Figure 9-4) relative to other simulations in the set and the new block-based capability 
acquisition policy [Rum02c], as an interim method pending a more definitive model 
[Gre02a]. In October 2002, the MDA Model and Simulation Working Group (MSWG), 
composed of members from all the Agency program offices, MDA directorates and execut¬ 
ing agents established a fidelity technical working group (TWG) with the following char¬ 
ter: 

• Define M&S fidelity in the context of the BMDS System-level M&S, 

• Propose a time-phased schedule for implementing BMD fidelity standards, 

• Determine the requirement for multi-resolution capabilities [Gre02b]. 
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Figure 9-4. MDS System-Level Model and Simulation Capability (After [TEM03]) 
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One of the early produet deliverables from the BMDS MSWG Fidelity teehnieal 
working group (Figure 9-5) provides the methodology the group seleeted to support their 
tasks. The TWG’s initial effort [Gre02b] addressed the BMDS kill-chain detection func¬ 
tion, sub-functions (e.g., search, detect, initiate track, track) and the respective fidelity at¬ 
tributes of the BMD sensor family of radar detection models as they relate to the BMDS 
System Capability Specification, a major BMD system engineering specification. The 
[WAD-l-02] approach also addresses another major Department systemic deficiency where 
major M&S initiatives and the Department’s system engineering processes lack a clear in¬ 
tegrated relationship, linkage, and common focus. 
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Figure 9-5. Example of Radar Fidelity in Core Models (From [TEM03]) 
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3, BMD System-Level M«&S Radar Detection Models 

There are several eritieal funetions and sub-funetions identified in the BMDS kill- 
chain framework (see Figure 9-2) required for the successful execution of the BMDS mis¬ 
sion. However, in this dissertation, we address only a subset of the detection function at a 
very high level to compare and contrast the development designs for radar detection models 
in the five BMDS system-level simulations. These radar model representations form pat¬ 
terns used later as the foundation for proposals on product lines and components. 

The CAPS simulation, considered “low” fidelity, provides Radar Detection Models 
with three basic detection capabilities: 

• Constant range, referred to as “cookie cutter”, 

• Scaled equation with reference parameters, 

• Full range equation [SpaOO] 

CAPS also provides various fidelities supported by four different threshold options: 

• Deterministic, using a standard signal-to-noise threshold defined by the user, 

• Probabilistic, using a random probability of detection threshold, 

• Deterministic, using a probability of detection (Pd) defined by the user, 

• Deterministic, with cumulative Pd threshold set by the user [SpaOO]. 

The optional degrees of flexible model methods and fidelity may appear as an unexpected 
capability in a highly aggregated simulation of CAPS class, except the detect function of 
the BMD kill chain is critical to the credibility of all BMD system-level M&S. Parameters 
[SpaOl] maintained in database files support most CAPS functions. 

The MDWAR simulation, considered “low-medium” fidelity, employs a generic ra¬ 
dar sensor model for detecting and tracking simulated threat targets, integrated with the 
Parallel Discrete Event Simulation (PDES) engine [TRW02a]. The model supports all the 
events needed to create a generic sensor model and certain specific radars, with design con¬ 
siderations driven by a need for a fast generic sensor simulation modeling components 
common to all types of sensors [TRWOOb]. The MDWAR real-time radar simulator, op¬ 
timized for large, real-time C2 wargame simulations may be configured to represent a num¬ 
ber of different radars, with a design goal of tracking 1,200 air breathing threats (e.g., air¬ 
craft, cruise missiles) with 100 radars while maintaining wall-clock time [TRW02b]. 
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The MDWAR design objectives support the requirement to provide users with real¬ 
time radar simulations driving fifty to seventy-five player/observer graphic workstations in 
very large human-in-the-loop wargame scenarios [TRW02b]. In order to achieve the re¬ 
quired performance, the radar sensor model depends on the following assumptions: 

• All sensor fidelity enhancements must be selectable by .par (e.g., parameter files) so 
they can be switched off if not required, 

• MDWAR models perfect communications between the sensor and the battle man¬ 
ager, as opposed to perception-based methods used in other simulations, 

• MDWAR models perfect weather without adverse conditions that may add to the 
fog and friction of war, 

• MDWAR does not model chaff clouds or jammers [TRW02b]. 

MDWAR does not model radar resolution, but provides perfect radar resolution in 
which every truth target, if detected, generates a radar return. However, discrimination is 
imperfect, generated by a time-based probability model [TRW02b]. The radar sensor 
model has unlimited sensor resources, except real-time constraints for keeping up with the 
wall-clock time; employs radar cross-section (RCS) data from a classified data table based 
on target type, radar frequency, and polarization; employs a Swerling target model (e.g., 
cases 0,1,2,3,4); and computes Pd on a random draw to determine if detection happens or 
not [TOM99, TRW02b]. MDWAR uses truth-based algorithms, versus a probability model 
[AGJ95] to determine a hit or miss for kill assessment [TRW02]. 

EADSIM, considered “medium” fidelity, also provides significant flexibility and 
options for radar representations, including complex models supporting both deterministic 
and probabilistic radar range detection [TOM97]. Signature modeling is activity dependent 
and includes user-specified functions for frequency, wavelength, aspect angle, with the user 
specifying the granularity of the tabular data [TOM97, TBE02a]. The radar sensor model 
may use a radar range equation, with jamming to detect targets, with the radar sensor class 
using a detection probability employing signal-to-noise (SNR) ratio, RCS, and bum- 
through calculations [TBE02b]. 

EADSIM employs a detection gate criteria as a means to model sensor detection 
limitations with a four gate criteria: 

• Sensor-to-target range, 

• Absolute target speed. 
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• Sensor-to-target range rate, 

• Target altitude [TBE02b]. 

The four gate eriteria share eommon test criteria with a region defined by minimum and 
maximum values, and no detections possible outside the defined region [TBE02b]. EAD- 
SIM defines RCS in one of three ways: uniform RCS, a single value specified for a system 
or weapon type; roll symmetric RCS tables, defined by frequency; or laterally symmetric 
RCS tables, normally employed with aircraft and cruise missiles, when the right and left 
sides of the system are mirror images [TBE02b]. 

EADSIM supports radar sensors at different levels of fidelity including a simple 
sensor approximating the to-level performance of the radar with basic search, detect, and 
track functions; a compound sensor supporting multifunction radars and advanced func¬ 
tions including multiple autonomous search sectors, cued search, track, interceptor support 
functions, resource management, sensor constraints and kill assessment [TOM97]. In 
EADSIM, target detection includes specific sensor-to-target detection tests including: 

• There is an unobstructed view of the target, 

• There is an unobstructed line-of-sight between the target and the sensor, 

• The target is in the sensor’s field of view [TOM97]. 

Specific detection tests may be either deterministic comparing computed signal-to- 
interference ratios with a user-specified variable, or probabilistic with Swerling target fluc¬ 
tuation models providing a computed probability of detection with a random draw to de¬ 
termine detection [TOM97, TBE02b]. Both methods support the effects of clutter, multi- 
path, atmospheric attenuation, and jamming on the radar sensor [TB02b]. In all cases 
EADSIM employs digital terrain models to check for terrain masking and inter-visibility 
issues [TOM97]. 

EADTB, considered a “medium-high” fidelity simulation, supports sensor models, 
which detect objects through the measurement of energy, reflected or emitted by those ob¬ 
jects (e.g., the sensed platform supplies the signature, and not the sensing platform). The 
sensed object provides the signature on request based on the sensing geometry [RayOlb]. 
The user defines sensor performance (e.g., power, gain, target signature characteristics, en- 
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vironmental effects) input variables, which the sensor algorithm computes to detect targets 
[Ray02a]. 

Although EADTB models both active and passive sensors including electro-optic / 
infrared passive sensors, and RF passive sensors, radar detection models are active sensors 
that transmit radio frequency (RF) energy and measure its reflection from the target body 
[Ray02a]. FADTB sensor models include separate methods for detection and tracking and 
may include: 

• Scanning, 

• Jamming, 

• Fine-of-sight obscuration, 

• Antenna patterns, 

• Sensitivity and processing, 

• Sensor beam or field of view [RayOlb]. 

The level of detail in an FADTB sensor model varies from a functional sensor in¬ 
corporating range, target type, and search volume; to a high-detail sensor with many input 
sensors to model gain patterns, polarization, Doppler spreads, ambiguous ranges, pulse 
compensation ratios, and main- or side-lobe jamming [Ray02a]. The modular sensor 
model includes four subcomponents: scanner, transmitter, receiver, and data processor and 
the management of these resources, while the radar model simulates different radar func¬ 
tions (e.g., search, track, acquisition) and different types of radars (e.g., mechanical, phased 
array) [Ray02a]. 

EADTB computes the probability of detection as a function of the probability of de¬ 
tection versus SNR distribution and the difference between the SNR and the detection 
threshold, followed by a random draw to determine target detection [Ray02a]. EADTB 
sensors operate in explicitly defined modes supporting multiple scan modes and track 
modes and accommodate three classes of data: static, dynamic, and mode-dependent 
[BBB-l-96]. In addition, EADTB may model different RF phenomena, with over 180 types 
of radar data input possible. 

MDSE currently tests the BMDS at the FoS level (e.g., interoperability), as opposed 
to the system or subsystem level, which establishes the required level of fidelity and accu¬ 
racy for the three embedded model categories: BMC2, launcher and fly-out models, and 
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sensor front end models [McQ97]. The MDSE sensor front-end model provides accurate 
detection, tracking positions, and timelines with the initial detect function of the kill chain 
required to be accurate within +/- 10% [Too94]. MDSE provides the RCS for missiles as a 
function of aspect angle (e.g., a measurement from missile nose to tail), required for initial 
detection and tracking results [McQ97]. MDSE sensor models generate realistic track posi¬ 
tion fluctuations for covariance matrices supporting the development of Tactical Digital 
Information Eink (TADIE) message standard [DoD97] J3.6 messages. Although MDSE 
radar models are not detailed enough to perform the classification or discrimination func¬ 
tions, they are sufficient for use in determining the status or reporting responsibility (R2) 
message traffic [Too94]. 

K, BMD SYSTEM-LEVEL M&S REPRESENTATION HETEROGENITY 


A review of Table 9-1 through Table 9-16 reveals several specific contributing 
causes for M&S representation heterogeneity issues in the five BMDS System-Level M&S, 
and possible contributing causes for M&S representation heterogeneity issues in the De¬ 
partment’s large-scale legacy M&S including: 

• The absence of a specific system representation for existing systems within a single 
simulation, 

• The absence of a specific system representation for existing systems within a se¬ 
lected set of federated simulations, 

• The absence of specific system representations for different configurations of the 
same existing system within the same simulation (e.g., THAAD PDRR system and 
the THAAD EMD system), 

• The absence of specific system representations in the selected set of federated simu¬ 
lations for the same existing system/platform (e.g.. Aegis Weapon System) with dif¬ 
ferent or unique configurations (e.g., SM-3 Block 1 A, Navy Theater Wide (NTW), 
and Navy LEAP ALI), 

• Temporal heterogeneity- 

o CAPS is a discrete-event simulation and a hybrid event-driven, time-stepped 
constructive simulation [SpaOO]. In the hybrid event-driven, time-stepped 
constructive simulation mode, time-compression algorithms reduce run time 
in scenarios with several periods of intense activity, separated by long peri¬ 
ods of relative inactivity prior to the next event. Using time compression, 
CAPS is able to run an 80-day theater-level campaign in approximately an 
hour, depending on the available machine resources. 
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o EADSIM is both event-driven and time-stepped, in a hybrid approaeh where 
the C3I proeesses are event driven and all other processes are time-stepped, 

• Geodetic heterogeneity inconsistencies cited in Table 9-15 are a contributing cause 
of heterogeneous system representation anomalies, including substantive interop¬ 
erability discussed in Chapter IV. The requirement to accurately model the BMD 
on worldwide basis dictates a standard geodetic environment, accurately established 
in all System-Level M&S. 

L, HMDS SYSTEM-LEVEL M&S DEVLOPMENT DEMOGRAPHICS 

Table 9-18 provides the development demographics for the five BMD System- 
Level simulations. Official development for all five simulations averaged slightly over 
three years and ranged from two to five years for the first software release. MDA execu¬ 
tive agents^^"^ in Huntsville, AL, and Colorado Springs, CO manage four of the systems de¬ 
veloped by contractors. The number of developers cited in Table 9-18 indicates the aver¬ 
age annual simulation software development level of effort (LOE) allocated for these sys¬ 
tems for the fiscal years 2000-2003, with the exception of EADSIM. The EADSIM pro¬ 
gram manager determines requirements and accepts resources from over three hundred or¬ 
ganizations. Only three of the thirty-six EADSIM developers noted in Table 9-18 directly 
support MDA requirements. 

The simulation software developer metrics for the missile domain M&S is an im¬ 
portant metric, as is the location of the major development efforts. The Fletcher Report 
[FMA+84] provided strong justification for mature software development organizations 
with strong missile domain expertise employing simulations as a design tool noting: 

Specifying, generating, testing, and maintaining the soft¬ 
ware for a battle management system will be a task that far 
exceeds in complexity and difficulty any that yet has been 
accomplished in the production of civil or military software 
systems [FMA+84]. 

The number of software developers and maintainers for the missile defense domain 
M&S software identified in Table 9-18 identify sources of the current domain expertise and 
market presence required to meet the software challenges cited by references earlier in the 
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chapter, and earlier missile defense studies ineluding the Fletcher Report [FMA+84], the 
follow-on Eastport Study Group report [CCL-l-85], and Congressional studies by the Office 
of Technology Assessment [SCC+88]. The co-loeation of the simulation software devel¬ 
opers with the missile defense program offices and their software developers in Huntsville, 
AL., and with the developers of the battle management software in Colorado Springs, CO., 
supported by a distributed network of missile defense expertise play a critieal role in the 
M&S evolutionary aequisition strategy supporting the BMDS. 


BMD SYSTEM-LEVEL M&S DEMOGRAPHICS 


CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

START DEV 

1991 

1996 

1987 

1989 

1993 

RELEASED 

1994 

1999 

1989 

1994 

1996 

EXEC AGENT 

MDA 

MDA JNIC 

ARMY 

ARMY 

ARMY 

DEVELOPER 

SPARTA 

TRW/ 

TELEDYNE 

HUGHES / 

TELEDYNE 



RAYTHEON 

(TBE) 

RAYTHEON 

(TBE) 

STAFFING 

4 

83 

36 

65 

76 

LOCATION 

VA 

CO 

AL 

AL 

AL 

CMM LEVEL 

I 

3 

3 

1 

3 


ISpaOO, SpaOI, 

[CM99, 

lBAC+95, 

[BBG+96, 

[Too94, 


BBG+99f| 

TOM99, 

TOM97, 

RayOIa, Rayo2a, 

McQ97, 

Citation from 


TRWOOa, 

BBG+99e, 

RayOlb] 

TEMOO] 



TRWOOb, 

TBE02a, 





TRW02b] 

TBE02b] 




Table 9-18. BMD System-Level M&S Demographics 


The Department requires proeess maturity from the software acquirers and the 
software developers. The BMDS government personnel aequiring the simulation software 
products, must understand the missile defense system evolutionary aequisition strategy 
[Ald02d] and also develop and define the applieable methods (e.g.. Software Acquisition- 
Capability Maturity Model (SA-CMM), or a similar methodology to determine and report 
proeess adherence and performance metrics [AS03]. Similarly, the contractors serving as 
the builders and vendors for the core simulation software assets must have the software de¬ 
velopment and performance skills [Gan99] to understand and translate the underlying eom- 
plex scienee, engineering, and teehnology into the missile defense domain M&S software 
at a minimum of SEI CMM Level 3, or its equivalent. The SEI Capability Maturity Model 
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(CMM) assessment for three of the five software development organizations for MDWAR, 
EADSIM, and MDSE is CMM level 3, while a fourth developer for EADTB is pursuing 
ISO 9000 certification. 

M, ANALYSIS OF BMDS M&S SOFTWARE STATE FACTORS 

Although these five simulations arrived at their current state by different circum¬ 
stances and development models, they share several common, important characteristics. 
First, as noted previously in the M&S synopsis for each simulation, these simulation tools 
meet a unique requirement (e.g., Human-in-the-loop, Hardware-in-the-loop, architecture 
development) for the development of the missile defense system. Second, there is a long 
lineage of experienced stakeholders, a distributed organizational cohort of users, develop¬ 
ers, warfighters who understand the underlying missile defense technologies, have devel¬ 
oped relationships within the community, interact often on studies, exercises, or analyses, 
and share the same common mission-focus. Third, these simulations produce results 
deemed credible enough for major Department decision-makers to support the authoriza¬ 
tion and allocations of continued Department investments for their evolutionary develop¬ 
ment. Fourth, these simulation systems represent the repositories of missile defense do¬ 
main expertise, intellectual capital, and/or presence in the marketplace. Fifth, these simula¬ 
tions are missile defense core assets and repositories for missile defense requirements, 
knowledge and insights that may not be well documented or understood elsewhere. 

Easily, in an analogous sense, these five legacy simulation systems and support en¬ 
vironments survived a Darwinian winnowing process as many other Department simulation 
systems atrophied, expired, or experienced a stasis of resources during the 1990s. The co¬ 
hort of organizational sponsors, users, collaborators, and developers for these five simula¬ 
tion systems shared a parallel sociological process as a distributed organizational constitu¬ 
ency and have at this point satisfied what Maslow would have identified as the physiologi¬ 
cal need of survival. The distributed organizational constituency or cohort of supporters, 
collaborators, and stakeholders, nonexistent when the simulations were first fielded, ex¬ 
panded their knowledge-base concurrently over time with the maturation of the simula¬ 
tions, and contributed significantly to the systems development and longevity There is an- 
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other important consideration. All five system-level simulation systems completed the dif¬ 
ficult early life cycle phases including the problematic requirements generation process and 
the continuous justification for funding. 

Working within the Departmental transformation process, and supporting an evolu¬ 
tionary block acquisition strategy, the missile defense domain system engineering process, 
simulation software engineering capability, supporting organizations, and processes need to 
concurrently mature in an extremely complex and technically demanding environment. 
Furthermore, when compared to less-complex systems, critical differences in the missile 
defense software quality requirements exist, which would; 

• Permit less opportunity for human intervention, 

• Have to handle more objects in its battle space, 

• Have to manage a larger battle space, 

• Use different weapons and sensor technology, 

• Contain vastly more elements, 

• Have more serious consequences of failure, 

• Have to operate in a nuclear environment, 

• Be under active attack by the enemy, and 

• Be useless if it failed catastrophically during its first battle [SCC+88]. 

A review of the Agency’s funding documents for the selected core M&S revealed 
an inconsistent funding profile before 2000 for the reasons noted earlier in the chapter. In 
addition, resolution of potential Year 2000 problems [Gre97, WG99] and the Department’s 
mandate for HLA interoperability expended funding resources earmarked for simulation 
development during the 1990’s [ISE99]. In the future the core M&S program appears as a 
relatively stable program according to the Department’s Future Years Defense Plan and the 
Program Objective Memorandum documents, with level and consistent funding levels pro¬ 
jected through 2007. However, with no new projected funding additions, future simulation 
software engineering improvements will compete for a finite set of the Agency’s resources. 

N, ANALYSIS OF HMDS M&S SOFTWARE DEVELOPMENT PORTFOLIO 

Table 9-19 identifies the programming languages and the associated source lines of 
code (SLOG) of the five software-intensive BMD System-Level Model and Simulations. 
The BMD System-Level M&S comprise a total of nearly 8.5 million SLOG. This corpus 
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of code is a domain capital/core asset of the BMD system portfolio. A few programming 
languages ineluding Ada, C, C++, and FORTRAN equate to approximately ninety-three 
pereent of the total SLOC. The Ada (28%) and C (32%) programming languages are sixty 
pereent of the total SLOC for the five simulations, a metric heavily influenced by the 2.25 
million SLOC of Ada eode resident in the EADTB simulation. The C or C++ languages 
are approximately fifty-five percent of the total portfolio SLOC. 

A review of Table 9-19 reveals less than two pereent of the Ada eode written by 
CMM level 3 software developers. Conversely, the three CMM level 3 software develop¬ 
ers identified in Table 9-19, wrote approximately 4.2 million SLOC, or nearly fifty pereent 
of the total 8.5 million SLOC in C or C++. In addition the C and C++ programming lan¬ 
guages are also the most eommonly used programming languages, with a signifieant pres- 
enee in four of the five simulations. 

Viewed from the simulation software perspective, MDSE (35%), EADTB (34%), 
and EADSIM (20%) eontain nearly ninety pereent of the total 8.5 million SLOC of the 
five-model portfolio, with EADTB and MDSE approaehing the three million SEOC mile¬ 
stones. CAPS, the smallest BMD System-Eevel M&S employs mostly EORTRAN, C and 
C++. The C++ and JAVA development environments support the MDWAR effort. EAD¬ 
SIM deseribed by [TOM97] as “objeet-like nature” and written primarily in C and C++, 
with some EORTRAN, has the highest population of other COTS and proprietary lan¬ 
guages. EADTB is an event-driven simulation written primarily in Ada, with some eom- 
ponents developed in C and JAVA, and several other languages. MDSE is the largest lan¬ 
guage development and maintenanee environment ineluding C++, C, EORTRAN, JAVA, 
and Ada, due to its distributed development environment. Oraele relational database sys¬ 
tems house MDSE and EADTB. 

In many respeets the development eyele of these five simulations listed in Table 9- 
19 mirror the Department’s software development experienee since 1985. CAPS and 
EADSIM were developed in an ineremental, evolutionary manner from an existing eapabil- 
ity at relatively low and ineonsistent funding levels, separate from the Department’s aequi- 
sition thresholds requiring the use of Ada. With the exeeption of the use of EORTRAN, 
these systems developed around the languages du jour, primarily C and C++. 


270 



Conversely, EADTB and MDSE were relatively large systems with the need to jus¬ 
tify significant amounts of Department appropriated funding, which triggered the imple¬ 
mentation of the Ada language. Approximately seventy-eight percent of EADTB remains 
in the Ada development environment, with a significant presence in C and a growing body 
of JAVA code. In MDSE the amount of Ada code equates to less than five percent of the 
MDSE system total, while C and C++ correspond to approximately sixty-five percent of 
the MDSE portfolio, followed by a twenty-eight percent EORTRAN population, the largest 
EORTRAN presence in the five simulations under study. 


BMDS SYSTEM-LEVEL M&S LANGUAGES / SLOG - 2002 


MDL/LANG 

ADA 

C 

C++ 

FORTRAN 

JAVA 

Other"" 

Total 

CAPS 


600 

650 

750 



2000 

MDWAR 



872,434 


47,335 


919,769 

EADSIM 


1,338,481 

99,705 

34,234 


225,933 

1,698,353 

EADTB 

2,250,000 

490,000 



80,000 

80,000 

2,900,000 

MDSE 

146,800 

891,600 

1,051,000 

837,300 

47,000 


2,973,700 

TOTAL 

2,396,800 

2,720,681 

2,023,789 

872,284 

174,335 

305,933 

8,493,822 


Table 9-19. 


BMD System-Level M&S SLOC - 2002 (After [TEM03]) 


O. BMDS M&S SOFTWARE INTEROPERABILITY 


BMDS System-Level M&S distributed interoperability capabilities includes HLA, 
DIS, and ALPS. Table 9-20 illustrates the distributed interoperability status of the five 
simulations under study. Lour of the five simulations, for an eighty percent compliance 
rate, are HLA-compliant, with MDSE, the agency hardware-in-the-loop tool as the sole ex¬ 
ception. A review of the simulations supporting the DIS standard also reveals an eighty 
percent compliance rate, with CAPS being the only simulation under review lacking DIS 
support. Prom the interoperability perspective, all five simulations are technically interop¬ 
erable with either the HLA or DIS protocols, and sixty percent of the simulations, including 
MDWAR, EADSIM, and EADTB are both HLA and DIS compliant. EADSIM is the only 
one of the five simulations supporting all three interoperability standards, while CAPS sup¬ 
ports only HLA. Overall, the systems under study achieved eighty percent for HLA and 
DIS interoperability, twenty percent for the ALPS standard, with an overall distributed 


Other languages include a mix of COTS and proprietary languages: Perl, SQL, HTML, C30, RSRC, H [TEM03] 
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simulation capability of sixty percent when eonsidering all three M&S distributed M&S 
protocol standards. 


BMDS SYSTEM-LEVEL M&S DISTRIBUTED SIMULATION CAPABILITY 

INTEROPERABILITY 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

% 

HLA 

Y 

Y 

Y 

Y 

N 

80% 

DIS 

N 

Y 

Y 

Y 

Y 

80% 

ALPS 

N 

N 

Y 

N 

N 

20% 

# Avail /Total/ % Avail 

1/3/33% 

2 / 3 / 66% 

3 / 3 /100% 

2 / 3 / 66% 

1/3/33% 

60% 


ISpaOO, 

[CM99, 

lBAC+95, 

[BBG+96, 

lToo94, 



SpaOl, 

TOM99, 

TOM97, 

RayOla, 

McQ97, 


Citation from 

BBG+99f| 
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Rayo2a, 
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TRWOOb, 

TBE02a, 
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TRW02b] 

TBE02b| 
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9-20. 


BMD System-Level M&S Interoperability Status 


P. EDUCATION ENABLERS SUPPORTING M&S CREDIBILITY 

A few years ago a television eommereial showed a seene where a company execu¬ 
tive, after reeeiving a presentation on the problems facing his company, asked the same 
team to help resolve the issues. In response the presenters noted they only identified prob¬ 
lems, and did not fix them. The Department’s history with resolving persistent systemic 
software issues is analogous to this scenario. The Department has done a thorough job 
identifying the many issues adversely affeeting simulation software eredibility. There have 
been many signifieant policies, programs, processes, and proeedures implemented to 
eounter these issues, although. Department simulation software still laeks overall eredibil- 
ity. 

One of the underlying reasons for a lack of credible simulations is the Department’s 
laek of depth, understanding, experienee, and maturity with diseiplined software engineer¬ 
ing. We believe a critical pre-condition for resolving this shorteoming is the need for eon- 
tinuing software engineering education [Tuc02] within the Department, aeknowledging the 
software engineering field is still eoalescing [Boe02, CT02, HH02a, HH02b, SBM02, 
SHM02, UHC02]. In a similar situation during the 1980’s the Department identified the 
need to improve the professional development of an industrial-age aequisition workforee 
and implemented the Defense Aequisition Workforee Improvement Act. Understanding 
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the changing, challenging dynamics represented by today’s information-age working envi¬ 
ronment, and the future demands of continuous transformation, the Department instituted a 
Continuous Learning Policy [Ald02f], acknowledging the need to operate as a continuous 
learning community, continually striving to improve professional knowledge and perform¬ 
ance in the Acquisition workforce. 

Today, the Nation has an information-based economy and the Department has an 
information-based national military strategy based on software. Unfortunately, there are 
few indications that the Department has developed the new capabilities to define, develop, 
acquire, and manage the myriad of complexities involved in software-intensive systems. 
The Department recently acknowledged this shortfall and developed a program [Ste02b] to 
enable selected members of the Department’s military and civilian workforce earn master 
or doctoral degrees in the relevant scientific, technical, and management academic disci¬ 
plines supporting information assurance. 

In a similar program. Dr. Patricia Sanders, the current MDA System Executive Of¬ 
ficer and former BMDO Deputy for Test, Simulation, and Evaluation, initiated the BMDO 
Software Engineering Improvement Program in 2000, in conjunction with the Naval Post¬ 
graduate School’s (NPS) Software Engineering Distance Eearning program, pioneered by 
Professor Euqi. The NPS Software Engineering Distance Eearning program provides 
Software Engineering programs at the master and doctoral level, for Department software 
practitioners to study, research, and advance software engineering principles and technol¬ 
ogy vital to Department researchers and program managers [NPS02]. The BMDO Soft¬ 
ware Engineering Improvement Program complemented the Agency’s Acquisition Certifi¬ 
cation Program supporting employee Eevel III Defense Acquisition Corps certification 
within eighteen months of arrival and an eighty-hour continuous learning program for the 
Eevel III workforce every two years. 

The melding of information technology employed in the NPS distance-learning 
model, with the collaborative ability of a distributed learning process shared by multiple 
major Department acquisition programs in different domains located around the Nation 
(e.g., the Army’s Tank and Automotive Command, (TAACOM) in Michigan, the Navy’s 
Space and Warfare Command (SPAWAR) in California, and the Missile Defense Agency 
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(MDA) in Washington, DC), employing the adult, eooperative learning model (e.g., study- 
and work-related researeh), working elosely with and mentored by the NFS Software Engi¬ 
neering faculty appears to be exactly the type of education Transformation model the De¬ 
partment needs. Software engineering research supported by the NFS Distance Learning 
model also has a reciprocating function, continually updating the academic community 
with real-world research issues, and providing a source for the latest research to resolve 
current Department software engineering challenges. A recent example of this successful 
paradigm was the publication of an NFS Software Engineering Distance Learning Master’s 
thesis Conceptual Framework Approach for Systems-of-Systems Software Developments 
[CafD3] in support of the BMDS evolutionary acquisition strategy^^^. 

Q. SUMMARY 

Chapter IX provided an overview of the BMDS M&S domain and included domain 
background, the domain M&S hierarchy, a BMDS System-Level M&S synopsis, M&S 
demographics, and a review of the missile defense system representations populating the 
five BMD System-Level simulations under study in this dissertation. The BMD domain 
overview section provided the Agency background and highlighted the significant role De¬ 
partment organizational changes played in the current status of BMD System-Level M&S. 

Building on the organizational outline and M&S domain overview, the chapter con¬ 
tinued with a status review on the domain implementation of the Department’s policies for 
establishing credibility, including a concise VV&A background for each of the BMD Sys¬ 
tem-Level M&S. Summaries of the other M&S in the domain M&S hierarchy provided 
additional context for the analysis. A top-level review of the BMD System-Level M&S 
fidelity, and the foundations for radar sensor fidelity followed. 

The scope of this research included five large-scale. Missile Defense Agency Do¬ 
main legacy simulations. The research methodology supported by the NFS software engi¬ 
neering distance-learning model facilitated the timely study of Department primary source 

The Missile Defense Agency received the Department of Defense 202 M&S Award in the Acquisition category for the MDA “Enter¬ 
prise Strategy for Modeling and Simulation” from Dr. Ronald Sega, Director of Defense Research and Engineering at a Pentagon cere¬ 
mony on September 29, 2003. The NPS distance learning software engineering program and software engineering research supported by 
the NPS Software Engineering Department faculty at Naval Postgraduate School, Monterey directly supported many of the MDA M&S 
program improvements cited in the award. 
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material for software-intensive legacy simulation systems. This case study also employed 
selected product line practice areas (Organization Management, Technical Management, 
Software Engineering) as a tailored framework for the missile defense domain analysis. 

The analysis identified additional root causes for heterogeneous anomalies (e.g., 
substantive interoperability issues) in the BMD System-Level M&S, although a primary 
cause was not established. Research indicated that a number of factors converged to create 
the conditions favorable for the development of heterogeneous system representation 
anomalies in Department simulations, affectively reducing credibility in the simulation or 
federation operations, and ultimately reducing confidence in the results. Results from the 
Agency five-simulation sample closely correlated to the Department’s experiences with 
establishing simulation and federation credibility identified in Chapters II through Chapter 
VI. 
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X, A SOFTWARE ARCHITECTURE-BASED PRODUCT LINE MODEL FOR 
SIMULATION MODEL REPRESENTATIONS 

A. INTRODUCTION 

Chapter X introduces the detailed research design for the Simulation Software Ar¬ 
chitecture-Based Product Line Model including five major elements supporting the method, 
specification, design, and implementation. The chapter provides an architectural analysis 
of the Department’s existing de facto M&S software architecture and introduces the: 

• Simulation Software Architecture fAXT^-Based Model as an abstract software ar¬ 
chitecture-based horizontal foundation supporting multiple viewpoints and views, 

• Simulation Software Architectural Framework (SSAF), a second vertical-slice^*’^ 
architectural component overlaying the SSA, which includes system and environ¬ 
ment components [ABB+02, GAOS], 

• Simulation Product Lines Architecture (SPLA) providing the framework to man¬ 
age the variability^*^^ [BBOO, BosOO, DMHOl, RSCOO, MNJ+02, CBB+03], features 
[BosOO, KLD02], and differences between products comprises the third element, 

• Simulation Software Architecture-Based Product Line Model Domain Metadata 
Repository provides the structure for the Domain Metadata Registry modeled from 
[IS079c] to ensure interoperability with Department, Federal Government, and pri¬ 
vate sector metadata registries, 

• Architecture Readiness Levels (ARL) to measure future architectural components 
and products developed from this methodology, is the fifth and final contribution 

The SSA and SSAF establish the architectural foundation for the SPLA, which sup¬ 
ports variability and extensibility of the architectural construct. We also adapted the archi¬ 
tectural description conceptual framework^^^ from [lEEOOB] into the SSA, SSAE, and 
SPLA models composing the Simulation Software Architecture-Based Product Line 
Model. 


The chapter also suggests a complementary Domain Integrated Product Develop¬ 
ment Team (DIPDT) approach for implementing the Simulation Software Architecture- 
Based Product Line Model, to reduce the cycle time for resolving the multiple dimensions 


In UML this is called a partition or a set of related classifiers or packages at the same level of abstraction or across layers in a layered 
architecture [UML99]. 

Variability refers to decisions that will be made by a member of the development team prior to system deployment [CBB+03]. 
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The conceptual framework is the term of reference for the architectural description and establishes the terms and concepts for the 
content and use of architectural descriptions [lEEOOb], 
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causing heterogeneous system representation anomalies, and improving federation interop¬ 
erability. 


B. THE INFLUENCE OF SOFTWARE ARCHITECTURE IN DEPARTMENT 

M&S 


The foeus of this research is to improve the quality, aeeuraey, and eonsistency of 
authoritative representations in Department M&S, a major element in addressing the eur- 
rent heterogeneous system representation anomalies (e.g., substantive interoperability), 
whieh erodes eredibility in the Department simulation proeess, and injeets doubt instead of 
eonfidenee in the simulation results supporting Departmental- and National-level deeision- 
makers. This researeh indieates that after fifty years of development the Department M&S 
domain has not aehieved the desired level of eredibility and eonfidenee in simulation model 
representations, in part due to a laek of data sharing and data standardization problems. 
There are many reasons eited below for this assessment including; 

• Few mechanisms for enabling global data aequisition and interchange, especially 
across domain application areas, 

• The lack of unique global identifiers for standard data elements, 

• Inadequate doeumentation of data element eharaeteristies restrieting the usefulness 
of automation to loeate, retrieve, and exchange data, 

• A need for uniform guidanee for identifying, describing, and developing data ele¬ 
ments, 

• Loeating and retrieving a partieular data element is diffieult or impossible, 

• The absenee of a universal means for organizing standard data elements, 

• The laek of inter-organization data standards, and to a lesser extent, the laek of in¬ 
tra-organization eornmon data standards, 

• A proliferation of eustomized data interehange representations, 

• Imprecise data definitions and descriptions, limiting reuse opportunities, or multiple 
uses of the data, 

• A laek of standard data elements impeding global implementation of Eleetronie 
Data Exehange (EDI) [IS079a]. 

In eontrast to the mature eomputer hardware-engineering seetor, the software engi¬ 
neering seetor of the eomputer software industry is still developing a eonsensus on a soft¬ 
ware engineering body of knowledge [Moo99] and agreement on the benefits of software 
proeess improvement and definitions of software arehitecture. Although standards exist for 
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three of the four components needed for open information processing systems (e.g., hard¬ 
ware, software, and communications), standards for the fourth component of open informa¬ 
tion processing systems, data specification, is still under development by the international 
standards community and a relatively new member of the Open Systems Interconnection 
Environment [IS079a]. In addition, there are many other salient issues including informa¬ 
tion operations, information assurance, and the security of personal information from mis¬ 
use for Department software engineers and software architects to consider. 

1. The Department’s De Facto M&S Software Architecture 

Software architecture, even if it is a de facto software architecture, strongly influ¬ 
ences a system over its life cycle. While computer hardware-related architectures sup¬ 
ported by Moore’s Law were ascendant over the past fifty years, software-related architec¬ 
tural considerations, if they existed, were of secondary importance to the overall system 
engineering and development process during much of the period. However, the cost and 
complexity of software-intensive systems changed the industry’s dynamics, as reflected by 
the publication of software architectural styles [SC96, BK99, HHPOO, BosOO, MMOl, 
CBB+OS, Fra03] descriptions [lEEOOb], and patterns [Fow97] for software-intensive pro¬ 
jects provided new views and theories on software architectural principles to the software 
engineering community. However, the emerging software architecture concepts currently 
affect little of the Department’s M&S software life cycle management practices for large- 
scale, legacy, software-intensive systems and simulation model representations. 

2, A Layered Architecture Model Approach 

The Department’s M&S Hierarchy provides the standard view of M&S today. 
However, the current M&S Hierarchy view lacks integration, treats abstraction inconsis¬ 
tently, lacks quantitative measurements, and is more aptly considered a simulation heuris¬ 
tic. From a different view, the current de facto Department M&S software architecture 
view appears closely aligned to a layered style . Although there are several software 
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A view is a representation of a set of system elements and the relationships associated with them [CBB+03]. [CBB+03] also viws 
each layer as a virtual machine with constraints on the relationships among the virtual machines. 
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architectural styles and descriptions cited in Chapter VIII, layering supports the principle of 
information hiding, theoretically providing a capability to change a lower layer behind the 
interface and not impact the layers above it. 

The Department's M&S Hierarchy continually evolved to support the changing re¬ 
quirements dictated by new strategies, evolving doctrines, funding justification, changing 
organizations, new weapon systems, and the growing demand for credible simulations. It 
is clear from the research that the Department’s M&S Hierarchy provided a commonly ac¬ 
cepted way of providing a high-level communication vehicle to convey an imprecise quali¬ 
tative definition of a certain level of simulation fidelity (e.g., engineering, engagement, 
mission, campaign). However, the Department's M&S Hierarchy is not an architectural 
construct and lacks any clear definition of the individual layers or the interface mechanism 
between layers. 

In addition, the hierarchical structure inaccurately implies that the higher layers of 
the Department's M&S Hierarchy build on the lower layer(s) in a logical, well-engineered 
manner. Moreover, since the Department's M&S Hierarchy lacks an architectural connec¬ 
tion with the many simulations it represents, changes to either the simulations or the hierar¬ 
chy lack synchronization or cohesion. Lastly, without an architectural framework, the cur¬ 
rent Department's M&S Hierarchy appears extremely limited in its ability to meet future 
requirements for component-based development and rapidly composable simulations. 

More specifically, the de facto Department M&S software architecture view ap¬ 
pears to have two layers,the Conceptual Layer and the Implementation Layer, illustrated in 
informal notation in Figure 10-1. The Conceptual Layer theoretically supports and main¬ 
tains the validated CMMS or FDMS conceptual models described in Chapter V. The Im¬ 
plementation Layer is the software implementation and deployment of the M&S system, 
theoretically verified against the validated conceptual model of the real world under study. 


Layering reflects a division of software into units, with each layer representing a unit [CBB+03]. [CBB+03] also views each layer as 
a virtual machine with constraints on the relationships among the virtual machines (e.g., an abstract computing device). 
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-► 

Implementation Layer 



Verifies 

Conceptual Layer 


The Real World - Layer 0 


Figure 10-1. Department’s De Facto Two-Layer M&S Architeeture 

In reality, at Department and the Ageney level, the theoretieal development of vali¬ 
dated eoneeptual models representing the first abstraetion of the real world, and the subse¬ 
quent verifieation of the simulation software implementation with the validated eoneeptual 
model oecurred more often in literature than in praetiee as noted in the Department’s V&V 
experienee cited in Chapter 111 through Chapter VI; and the Agency case study in Chapter 
IX. In addition, the development of a validated conceptual model and verified software 
implementation indicates a high level of coupling, which may hinder the Department’s ob¬ 
jective for greater reuse of components and composable M&S frameworks. 
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C. THE HORIZONTALLY-LAYERED SIMULATION SOFTWARE ARCHI¬ 
TECTURE 

1. The Software Architecture Layered Style 

The first element of the Simulation Software Arehitecture-Based Product Line 
Model is the Simulation Software Architecture (SSA). The proposed simulation software 
architecture style is the layered style. The layered view of the Department’s simulation 
software architecture, graphically presented as a layer diagram, segments software into 
each layer with constraints, connected by a public interface. Properly developed layered- 
style diagrams support the development of software-intensive simulation systems featuring 
portability, interoperability, and modifiability facilitating reuse, component-based devel¬ 
opment, and future composability research initiatives. The layered diagram sees common 
use, although some software architects employ poorly constructed layered diagrams, often 
using facilities of a higher layer without restrictions. 

Properly constructed layered architecture diagrams share a basic quality in which 
the layers interact according to a strict ordering relationship. Layered diagrams employ an 

.27.2 

allowed to use relationship. In the proposed simulation software architecture (SSA), the 
Referent Layer is beneath the Conceptual Layer and provides the views and viewpoints into 
the real world (e.g.. Layer 0). The implementation of the Conceptual Layer may use all of 
the public facilities provided by the Referent Layer through the interface. The Conceptual 
Layer is beneath the Component Layer. The implementation of the Component Layer may 
use all of the public facilities provided by the Conceptual Layer through the interface. The 
Component Layer is beneath the Implementation Layer. The Implementation Layer may 
use all of the public facilities provided by the Component Layer through the interface. 

We reviewed several other architectural styles as candidates for the SSA, including 
pipes and filters [SC96], layers [BBG+00], blackboards [BosOO], object-orientation, dis¬ 
tributed and systems, component and connector, interactive systems and several other 
styles described in Chapter VIII. [BosOO] describes this activity as imposing an architec- 
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The allowed to use is a variant of the depends-on relation, where if Pi uses P 2 , Pi sCorrectness depends on the correct implementation 
of P 2 [CBB+03]. 
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tural style on the software arehiteeture, requiring eareful eonsideration in the proeess and 
the impaet of a eomplete reorganization of the software arehiteeture. The primary goals of 
the SSA are to provide an arehiteetural framework to support eonstantly ehanging require¬ 
ments and seeondly, reduee simulation software eomplexity. Quality attributes support 
eaeh SSA layer. Figure 10-2 illustrates the proposed SSA model. 

In addition the review eneompassed the deeomposition style, uses style (e.g., de- 
pends-on relationship), generalization style (e.g., is-a relationship), and the layered style 
[CBB+OS]. The seleeted staek layer model for the SSA uses geometrie adjaeeney to repre¬ 
sent the allowed-to-use protoeol, as opposed to symbols (e.g., arrows, lines, diamonds). 
Other informal layered notations evaluated for the SSA layered style ineluded segmented 
layers, rings, three-dimensional models, and layers with sidecars [CBB+OS], with which we 
compared and contrasted the architectural attributes against the informal stack notation 
which represents the layers as a stack of rectangles. 

The SSA restricts upward communication by a lower layer of the architecture with 
the facilities of higher layers to ensure the validity of the desired architectural attributes and 
to retain the required properties to support the V&V process. Each horizontal slice or layer 
of the stack supports a vertical slice or abstraction of the product line, discussed later in the 
chapter, and no two layers contain the same product line abstraction. The thick horizontal 
lines between the layers indicate the constraints of interlayer communication to the inter¬ 
face protocols at the layer interface, with no other communications to the internal facilities 
of any other layer. 


283 





Implementation Layer 



Component Layer 



Conceptual Layer 



Referent Layer 


The Real World - Layer 0 


Figure 10-2. Simulation Software Arehitecture (SSA) Model 

2. Viewpoints and Views 

A viewpoint provides the pattern or framework, based on the stakeholders’ intended 
use of the system, for speeifying the view design and development. Viewpoints seleeted 
for the simulation software arehitecture description (SimSAD) may originate from the de¬ 
velopment effort or defined elsewhere and reused in the SimSAD, termed a library view by 
[lEEOOB]. Stakeholders select one or more viewpoints for the architectural description of 
the system supporting the intended use of the simulation. A viewpoint sets the modeling 
methods, procedures and analysis techniques for analyzing, creating, and displaying a view 
representation. The minimum specification for viewpoint metadata maintained in the 
Simulation Software Architecture-Based Product Eine Model Domain Metadata Repository 
includes: 

• The viewpoint name, 

• Eist of stakeholders supported by the viewpoint. 
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• The intended use, mission, and eoncerns addressed by the viewpoint, 

• The language, modeling teehniques, analysis methods used to develop a 
view based on the viewpoint, 

• The source for the viewpoint or library viewpoint including data sources 
for validation, 

• Formal or informal consistency and completeness tests applied to the 
models used to develop a view, 

• Evaluation or analysis techniques to be applied to the models, 

• Patterns, heuristics, and other guidelines supporting the development of 
a view [lEEOOb]. 

We define a view as a representation of a whole system or set of system elements 
from the perspective of a related set of concerns and the relationships associated with them. 
The view addresses stakeholders’ mission or intended use of the system in the simulation 
or federation. A view may also include one or more architectural models, which may sup¬ 
port one or more views, derived from another associated viewpoint architecture. 

Viewpoints and views are major contributors to this dissertation. Take the example 
of a house. The electrician, carpenter, plumber, bricklayer, HVAC (e.g., heating, ventila¬ 
tion, and cooling) contractor, cement contactor, and landscape specialist all have different 
viewpoints of the same house. All these viewpoints are correct since they support the mis¬ 
sion or intended use for the rules, procedures, and guidelines (e.g., appropriate building 
codes) the different craftsmen must follow to develop the different views of the architect, 
surveyor, builder, and owner. Different views are also important, since the builder wants 
the potential owner to buy the house, and later on the homeowner hopes the tax appraiser 
takes a lower-valued view of the property, while potential new buyers take a higher-valued 
view. 


3. The Referent Layer 

The Referent Eayer is the architectural framework of the SSA supporting view¬ 
points. Previous authors cited the need for a referent as a commonly understood standard 
[Gro99, RGHOO], or a properly developed conceptual model supporting fidelity [GET98, 
PacOlb] and validation [PacOlb]. We view system referents as software architecture con¬ 
structs, and components of the SimSAD, populating the first layer of the SSA, the Referent 
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layer. The single SSA Referent Layer in Figure 10-3 replaces all levels of the Depart- 
ment‘s M&S Flierarchy discussed in Chapter II (see Figure 2-2) and employs viewpoints of 
the real world as portals into the architectural framework. Viewpoints are new to the De¬ 
partment’s simulation software architecture design process. Viewpoints represent a one-to- 
one mapping from the real world into the architectural sink, the system referent, maintained 
in the referent Layer. Referents support one or more real-world viewpoints. 



Figure 10-3. SSA Referent Layer Model 

The Referent Layer performs the transaction manager role for the SSA. The Refer¬ 
ent Layer continually adds new viewpoints to the referent, while existing viewpoints 
change, evolve, and merge to new meet requirements, until archived pending future use. 
The Referent Layer provides the first interface to the services provided by the Simulation 
Software Architecture-Based Product Line Model Metadata Repository. This process ap¬ 
plies to all data sources including new system requirements and the core asset development 
methods discussed later in the chapter, including mining and reverse engineering. 
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4. 


The Conceptual Layer 


The Conceptual Layer in Figure 10-4 houses the conceptual views of the systems. 
Where the Referent Layer provided the transaction manager role for the SSA, the Concep¬ 
tual Layer provides the facilities for conceptual composition. The major component of the 
Conceptual Layer is the Simulation System Architectural Description (SimSAD). Each 
SimSAD comprises a specific system, which may include product lines, product families, 
complete enterprises, aggregated entities, subsystems, components, systems, and systems 
of systems, and their interfaces inhabiting an environment. 



Figure 10-4. SSA Conceptual Layer Model 

The Conceptual Layer provides facilities to “pull” viewpoints and referents from 
the Simulation Software Architecture-Based Product Line Model Metadata Repository pro¬ 
vided by the Referent Layer to support the development of views for the SimSAD and con¬ 
ceptual model construction. Conceptual views support the development of specific concep¬ 
tual models designated for an intended use or mission; or common-use conceptual models; 
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or conceptual models specifically designed for large-scale reuse. The Simulation Software 
Architecture-Based Product Line Model Metadata Repository provides configuration con¬ 
trol services for all views in the Conceptual Layer, and provides the interface control with 
the next higher layer, the Component Layer. 

5, The Component Layer 

The Component Layer in Figure 10-5 houses and manages the component views. 
Where the Conceptual Layer provided the facilities for architectural conceptual composi¬ 
tion, the Component Layer provides facilities to “pull” views from the Simulation Software 
Architecture-Based Product Line Model Metadata Repository provided by the Conceptual 
Layer to support the development of design strategies for component models. 



Referent Layer 


Figure 10-5. SSA Component Layer Model 

Conceptual views support the development of patterns of interaction for specific compo¬ 
nent models designated for an intended use or mission; or common-use component models; 
or component models specifically designed for large-scale reuse. 
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The Component Layer provides the arehitectural foundation for future eomposable 
item development by providing standards and conventions for quality attributes 
[WBL+01], composition rules, and visibility of assets. Components were discussed in 
Chapter VIII. The Simulation Software Architecture-Based Product Line Model Metadata 
Repository provides configuration control services for all SimSAD components in the 
Component Layer, and provides the interface control with the next higher layer, the Im¬ 
plementation Layer, which “pulls” component views as required. The development of ap¬ 
plication components is beyond the scope of this research. 

6, The Implementation Layer 

The Implementation Layer in Figure 10-6 houses and maintains the complete Sim¬ 
SAD system view, and provides the visibility of the component views available to imple¬ 
mentation layer developers. Where the Conceptual Layer provided the facilities for con¬ 
ceptual composition, the Implementation Layer provides facilities to “pull” component 
views from the Simulation Software Architecture-Based Product Line Model Metadata Re¬ 
pository provided by the Component Layer to support the development of system model 
views. Component views support the development of specific system models designated 
for an intended use or mission; common-use system models; or system models specifically 
designed for large-scale reuse. 

The Implementation Layer provides the architectural foundation for future com- 
posable system development. The Simulation Software Architecture-Based Product Line 
Model Metadata Repository provides configuration control services for all systems in the 
Implementation Layer and provides the interface control with the next lower layer, the 
Component Layer, “pulling” component views when required. The development of com- 
posable application systems is beyond the scope of this research. 
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Conceptual Layer 


Figure 10-6. SSA Implementation Layer Model 

7, The Simulation Software Architecture Framework (SSAF) 

The second element of the Simulation Software Architecture-Based Product Line 
Model is the Simulation Software Architecture Framework (SSAF), which provides the 
system environment components. The SSAF in Figure 10-7 adds the vertical dimension of 
the system environment to the horizontal architectural layer presented in the SSA. The re¬ 
quirement for a vertical dimension is an unusual requirement since many Department soft¬ 
ware systems developed in the past have evolved in a vertical manner, often referred to as 
“stove-piped” systems (e.g., personnel, logistics, medical, command and control), and 
lacked horizontal integration and interoperability. Flowever, current M&S Flierarchy mod¬ 
els lacks structure to vertically integrate the system model representations in a consistent 
fashion. 

Although the Department’ simulation development history experienced a similar 
pattern, architecturally it developed a different growth pattern. The simulation framework 
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and its internal models presented a de facto self-eontained horizontal architecture layer, 
within the simulation framework (e.g., engine, scenarios) providing the horizontal integra¬ 
tion and interoperability. However, the Department’s M&S Hierarchy provided few verti¬ 
cal integration enablers supporting a cohesive process for managing quality attributes for 
the same system at different levels of the M&S Hierarchy. This systemic shortcoming ad¬ 
versely affected the Department’s MS credibility. The SSAF presents a method to improve 
the Department’s vertical integration of M&S. The system component of the SSAF sup- 
ports the SimSAD. The system environment , includes the physical world and the exter¬ 
nal objects, conditions, or processes influencing the behavior of the system in past, present, 
and future dimensions, real or imagined, but clearly defined and documented in the Sim¬ 
SAD. 

A system has one or more stakeholders, with interests, intended uses and con- 
cerns . SimSAD stakeholders include simulation developers, architects, simulation spon¬ 
sors, V&V agents, accreditation agents, and ultimately Agency-, Department-, and Na¬ 
tional-level decision-makers. A system design also incorporates the intended use, or ful- 
fillment of a mission in the system’s environment. We use the terms mission and in¬ 

tended use interchangeably. The architectural description component of the SimSAD in¬ 
cludes the compilation of products to document architectures [lEEOOb]. 
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The system’s environment or context may influence the system’s setting, including circumstances of operational, developmental, 
political and other influences affecting the system. The environment may include other systems’ direct or indirect interaction with the 
system of interest through deflned interfaces, and also defines the boundaries and scope for the system of interest {lEEOO], 
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Stakeholder Concerns include the system’s development, operation, or any aspect of the system deemed critical by the stakeholder, 

including quality attributes detailed in Chapter VI. 
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A mission involves the operation or use of system by its stakeholders to meet objectives developed for the systems intended use 
[lEEOOb]. 


291 




Layer 0 - The Real World 


Figure 10-7. Simulation Software Arehiteeture Framework (SSAF) 

Recall that all systems have a defined or de facto architecture. The SimSAD de¬ 
scribes the conceptual architecture of a system and supports the follow-on development of 
product line descriptions for particular discrete instantiations of that specific architecture. 
The SimSAD maintains one or more constituent views and viewpoints. In addition to the 
selected views and viewtypes the SimSAD maintains other information and documentation 
supporting verification, validation, and accreditation processes, including a system over¬ 
view, system context, product family relationships, URLs for system validation data, and 
maintenance information. [lEEOOb] provides a concise listing of uses of architectural de¬ 
scriptions, architectural description practices, and recommended documentation supporting 
a SimSAD, including the specification for a group of systems sharing a common set of fea¬ 
tures (e.g., product lines). The SimSAD also maintains records and analysis data of known 
architectural inconsistencies. 
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8. 


Software Architecture Reconstruction 


Software architecture reconstruction provides a mechanism for the effective reuse 
of software assets [KC97, CW98, ObrOla, ObrOlb, KovOl, OSV02]. Architecture reuse is 
the cornerstone practice of product line development, supported by three sources for archi¬ 
tecture recovery: 

• Source code, which is authoritative, but does not hold all the information, 

• Documentation, which is normally undependable, incomplete, or outdated, 

• Human experts, who may be dependable, but biased [CW98]. 

The SEI developed the Options Analysis for Reengineering (OAR) methodology 
[BSW+99, BSOO, CN02] for making technical, organizational, and programmatic reengi¬ 
neering decisions. The OAR methodology: 

• Combines architectural and reengineering views with defined mappings between 
the views, 

• Classifies and stratifies reengineering options/approaches into distinct layers with a 
mapping between layers, 

• Provides key information for making informed choices about the time and criteria 
for each option/approach, 

• Codifies technical and non-technical risks for each option/approach, 

• Relates organizational and programmatic factors to support a unified reengineering 
option/approach [BSW+99]. 

[BSW+99] indicate the reengineering process includes three basic steps to evolve 
an existing legacy into a new system in a disciplined evolutionary process employing the 
Horseshoe Model, in Figure 10-8, which combines reengineering and architectural views of 
software analysis and evolution including: 

• Analysis of the existing system, and extracting artifacts from source code, the archi¬ 
tecture recovery / conformance process, 

• The architectural transformation of the legacy systems logical descriptions into new 
improved descriptions, 

• Architecture-Based Development (ABD) of the new system based on the new 
logical descriptions to instantiate the desired architecture [BSW+99]. 

The Horseshoe Model [BSW+99] shown in Figure 10-8 includes three levels. 

These levels according to [BSW+99] consist of a code-structured representation, which in¬ 
cludes parsing and analysis of the source code, abstract syntax trees (ASTs), and flow 
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graphs at the first level, and a functional level representation describing the relationships of 
functions, data and files at the second level. At the third level, the Horseshoe Model illus¬ 
trates the concept level representing combined function and code level artifacts used as the 
base of new architectural components or concepts. 
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Figure 10-8. 


The Horseshoe Model Underlying OAR (From [BSW+99]) 


The source-code level transformations may employ string matching and replace¬ 
ment techniques, or code-structure transformations based on syntax tree-based transforma¬ 
tions. Functional level transformations go beyond code-level changes and are concerned 
primarily with the reworking of the functionality, which may include changing from a 
functional design to an object-oriented design. The final transformation process, at the ar¬ 
chitectural level, involves changes to the basic building blocks including the typed of com¬ 
ponents, patterns of interaction, control mechanism, functional allocation, and data, and 
normally requires the greatest amount of time and resources according [BSW+99]. 

Architecture recovery and reconstruction is an iterative and interactive labor- 
intensive effort, dependent upon the level of specification, documentation, dissemination 
and control of the legacy architecture artifacts [KOVOl, OSV02]. Architecture recovery 
methods vary from entirely manual reconstruction processes to tool-supported manual re¬ 
construction and semi-autonomous reconstruction techniques, and may include data min- 
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and employment of architecture description languages [OSV02]. Pattern analysis 
supports architecture recovery and reconstruction efforts [OSV02]. [Fow97] addresses two 
categories of patterns: 

• Analysis patterns are groups of concepts that represent a common construction in 
business modeling, relevant to one or many domains, 

• Supporting patterns are patterns in their own right and valuable on their own, al¬ 
though the primary purpose supports the use of analysis patterns, making them real 
[Fow97]. 

Manual mining activities recover source information for architecture activities sup¬ 
porting product line development from the source code, documentation, and human ex- 
perts, since automated tools are currently limited. [KOVOl, OSV02A] suggest the use 
and synthesis of existing several tools and techniques into a “workbench” support envi¬ 
ronment for software analysts reconstructing architectures. A significant number of tools 
support view extraction, and several tools support view fusing and reconstruction. 

[HBL97, KC97, ObrOla, ObrOlb, BBC+02] support architecture reconstruction 
with automated tools; however, [OSV02] note that tool use is limited by a system using 
several languages, or on cases where the binary code is available, but not the source code. 
The possible lack of source code is a real possibility in reconstruction efforts with commer¬ 
cial components and the vendors decline to provide the source code [OSV02]. 

D, THE VERTICALLY-SLICED SOFTWARE PRODUCT LINE ARCHITEC¬ 
TURE 

[BosOO] suggests three purposes for architecture: as an individual software system, 
as product-line architecture^^^, or as a standard architecture for component development. 
Research indicates that component-based technologies remain immature, suggesting that an 
organizationally sponsored (e.g., domain) product line-based approach to implementing 
simulation software architectures, may serve as a predecessor technology to a follow-on 
component-based development environment. 


Mining includes resurrecting and rehabilitating a piece of an existing software system to serve in a new system for which it was not 

originally intended [Nor03]. 
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One existing tool is Dali is an SEI tool for extracting architectural information from an implemented system [CW98] 

[BosOO] defines the use of a product-line architecture as the common architecture for a set of related products or systems developed 
by an organization. 
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[BosOO] also identified two faetors for the approaeh to initiating a produet line: evo¬ 
lutionary or revolutionary; and suggested two market-based deeisions to develop a new 
product line or apply a product line methodology to a legacy product. Risk plays a major 
role in the decision process, especially in mission-critical, high-risk Department domains. 
Employing an evolutionary product line approach for large-scale, software-intensive legacy 
simulation systems has a potential in the Department’s M&S domain to: 

• Minimize investment risk and risks to continuity of operations, 

• Support the development of authoritative representations, 

• Reduce federation issues caused by substantive interoperability problems, 

• Identify the level of quality in the component product, and not just the level of ma¬ 
turity in the processes supporting development, 

• Support improved W&A practices, 

• Provide metric/measured credibility of Department M&S supporting improved con¬ 
fidence levels, 

• Initiate the transition of Department M&S to a supportable Department software ar¬ 
chitecture, 

• Capitalize on the current core asset investment, 

• Improve component syntactical / semantic inconsistencies between Department 
domains, 

• Provide a foundation for an integrated data and object model ontology, 

• Provide a common open source for validated algorithms, 

• Reduce overall life cycle support costs. 

It is possible to implement a software product line architecture shown in Figure 10- 
9 at various defined levels—for example: enterprise, domain, functional, system, subsys¬ 
tem, or component level. A key step in the systems and software engineering process is the 
iterative allocation of system requirements to hardware and software. This is also true for 
software-intensive product lines since the system may include hardware (e.g., hardware in 
the loop) or live simulation entities. 
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Layer 0 - The Real World 


Figure 10-9. Software Product Line Architecture (SPLA) 


The domain-level system and software engineering process defines the software 

279 280 

product line standard and product specification . These documents provide the alio- 

281 282 

cated level of product line system abstraction and data abstraction , including accu- 
racy , precision and quality factors or quality attributes , allocated as requirements 
to systems, subsystems, sets, groups, units, components, assemblies, subassembly, and fi- 
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A standard that defines what constitutes completeness and acceptability of items that are used or produced, fonnally or informally, 
during the software engineering process [1EE90]. 

The product specification is a critical component of product lines, and is essential to support variability. (1) It specifies the design 
that production copies of a system or component must implement, or (2) a document that describes the characteristics of a planned or 
existing product for consideration by potential customers or users [IEE90]. 

Abstraction is (1) a view of the object that focuses on the infonnation relevant to a particular purpose and ignores the remainder of 
the information. (2) The process of formulating a view as in (1) [IEE90]. 

The process of extracting the essential characteristics of data by defining data types and their associated tunctional characteristics and 
disregarding representational detail [IEE90]. 

Accuracy in product lines is either (1) a qualitative assessment of correctness, or freedom from error, or (2) a quantitative measure on 

the magnitude of error [IEE90]. 
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Precision is the degree of exactness or discrimination with a quantity is stated [IEE90]. 

A quality factor is a management-oriented attribute of software that contributes to its quality [IEE98d]. 

A characteristic of software, or a generic term applying to quality factors, quality sub factors, or metric value [IEE98d]. See Chapter 
VI for a more detailed discussion on quality. 
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nally the part level eomposing the product line. Product line quality attributes establish the 
foundation for product line software quality metrics and product metrics , essential to 
maintaining control of product line variability. 

The software product line architecture also supports a methodology independent 
classification scheme^^^, classified scheme items^^*^, and classified components^^^ for sev- 
eral components of data elements including object classes , properties , representa¬ 
tions^^'^, value domains^^^, data element concepts, as well as actual data elements, including 
keywords , thesaurus terms , taxonomy , and ontology taxa illustrated in Figure 
10-10. Classification attributes^^^ associate various classification schemes with selected 
components of data elements [IS079b] including: 

• Classified component ID, 

• Classified component name. 

• Classification scheme type, 

• Classification scheme name, 

• Classification scheme version, 

• Classification scheme item type, 

• Classification scheme item value [IS079b]. 


A function whose inputs are software data and whose output is a single numerical value that can be interpreted as the degree to which 
the software possesses a given attribute that affects its quality [IEE98d]. 

A metric used to measure the characteristics of any intermediate or final product of the software development process [IEE98d]. 
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The arrangement or division of objects into groups based on characteristics that the objects have in common (e.g., origin, composi¬ 
tion, structure, application, and function) [IS079b]. 
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Classified scheme items are Components of content in a classification scheme [IS079b]. 
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A component of a data element that may be classified in one or more classification schemes [IS079b]. 
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An object class is a set of ideas, abstractions, or things in the real world that can be identified with explicit boundaries and meaning 

and whose properties and behavior follow the same rules [IEE97a]. 
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A property is a peculiarity common to all members of an object class [IS079a], 
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A representation describes how the data are represented, and if necessary, a unit of measure or a character set [IS079a]. 
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A value (e.g., data value) domain is a set of permissible values, which may be enumerated or expressed by a description. The value 
domain provides representation, but does not include the data element concept the values may be associated with nor what the value 
means [IS079c]. 
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Keywords are basic attributes applied to object classes, properties, representations, and data elements and include the definition, 
obligation, data type, and comment [IS079b]. 
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Thesaurus terms can be associated with data elements and data element concepts [IS079b]. 
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Taxonomy is a hierarchical organization of concepts (e.g., taxa) based on generalization/specialization and the mathematical notions 

of sets, subsets, and set membership [IS079b]. 
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Ontology is a network organization of taxa meant to provide a model of some portion of the world, and consists of theories about the 
sorts of objects, properties of objects, and relations among objects that are possible in that portion of the world [IS079b]. 

The taxa in taxonomies and ontologies may be related to classified data registration components including object class, property, 

registration class, and data element concept [IS079b]. 
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Classification attribute descriptions include name, definition, obligation, condition, data type, and comments [IS079b]. 
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Figure 10-10. Software Arehiteeture Classifieation (From [lS079o]) 

A software produet line approach may be employed at one or more levels, depend¬ 
ing on many factors such as the application domain, degree of commonality and feature 
variability with other systems / sub-systems / components, and availability of candidate 
(reusable) software assets. To illustrate these engineering concepts, consider a missile sys¬ 
tem in a simulation as an integrated system of subsystems and components performing cer¬ 
tain functions for the missile, such as navigation. There are many options, including the 
establishment of a product line program for a computational system used across a family of 
missiles, or a product line program established for the navigation subsystems supplied to 
many missiles. 

The best leverage of simulation software architecture-based product lines occurs 
when they share a common architecture employing quality components for consistent 
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products. Each level of the M&S software arehiteeture-based product line model would 
have diserete quantifiable performanee and quality attributes. 

1. Core Architecture Asset Development 

a. Mining for Product Line Core Architecture Assets 

Systemie software reuse foeuses on large-grained reusable assets sueh as 
software arehiteetures, proeesses, doeumentation, test eases, and eomponents, versus the 
previous efforts in software reuse that revolved around small-grained assets of software 
eode, whieh experienced only modest gains [BFG+00, CN02]. The system architecture, 
doeumentation, and eomponents are central eore assets used to evolve and build the soft¬ 
ware produet line. 

[BSOOO, BOSOO, CN02] propose existing assets offers an organization the 
potential to leverage all, or part, of its eumulative system investments, and represents a 
eritieal praetiee area supporting a software produet line initiative. However, [BSOOO, 
CN02] eite signifieant risks aehieving sueeess developing software produet lines. [BSOOO] 
attributes the high risks to the lack of documentation, the poorly maintained state of many 
existing systems, and the fact that many systems, initially developed for different purposes, 
laek support for current software engineering approaehes. Research indicates this is gener¬ 
ally true in the Department’s M&S domain. [BSOOO, CN02] identify four basie steps to 
suoeessfully mine assets: 1) preliminary information gathering, 2) making deeisions on 
whether to mine assets and whieh type of overall strategy to use, 3) obtaining detailed 
teehnieal understanding of existing software assets, and 4) rehabilitation of assets. 

In mining legaey assets, [BSOO, BOSOO] suggest the focus should be on 
mining speeifie legaey software and evaluating adaptation techniques for product line use, 
as opposed to either eode level transformation or reengineering the system entirely. [BSOO] 
suggests the foeus on arehiteetural eompatibility and interfaees involves the identifieation 
of large-grained legaey system funetionality, and mining blaek-box software elements 
available for adaptation or wrapping to serve as eore assets, or support arehiteetural extrae- 
tion and reeonstruetion. Tradeoffs happen eontinuously in the early mining phases to re- 
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solve technical challenges, while constantly evaluating legacy software assets for mining 
potential, and to determine the practicality and cost effectiveness of the mining operation. 

b. Reverse Engineering for Product Line Core Architecture Assets 

[Til98] introduces a reverse-engineering^®^ environment framework to im- 
prove program understanding based on a descriptive model with categories of support 
mechanism features established on a foundation of attributes. [Til98] identifies three 
groupings of support mechanism category: unaided browsing, leveraging corporate knowl¬ 
edge and experience, and computer-aided techniques including reverse engineering. In ad¬ 
dition, program understanding according to [Til98] includes the identification, manipula¬ 
tion, and exploration of artifacts of a “particular representation of a subject system via men¬ 
tal pattern recognition by the software engineer and the aggregation of these artifacts to 
form more abstract system representations” [Til98]. 

Unaided browsing techniques normally employed in the process include re¬ 
viewing the source code [HCM02]; however, this method becomes unwieldy as the lines of 
code exceed normal limits of manual comprehension. Leveraging corporate knowledge 
and experience is also a valuable method, although maybe of limited value under certain 
conditions including: 

• The lack of available primary source corporate knowledge, 

• Third-party system acquisitions, 

• Outsourcing. 

Computer-aided reverse engineering methodologies is a third method of 
addressing some of the shortcomings of the two previous support mechanisms categories 
addressed by [Til98]. [JBL97, SLB-l-98, and BLS+99] contributed a body of knowledge 
germane to this research initiative with research on the Janus model at the Naval Post¬ 
graduate School (NPS). Follow-on NPS reverse-engineering initiatives including research 
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Reverse engineering is the process of understanding, analyzing, and abstracting the system to a form at a higher level of abstraction 
[01s95]. 

303 

Program understanding is a relatively new area of study with an evolving terminology and focus, with an objective to acquire suffi¬ 
cient knowledge about a software system so that it can evolve in a disciplined manner. The essence of program understanding is to iden¬ 
tify artifacts and understand the relationships, and is analogous to pattern matching concepts at various abstraction levels [Til98]. 
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A reverse engineering environment can manage the complexities of program understanding by helping the software engineer extract 
high-level information from low-level artifacts, such as source code, reduces the level of tedious, error-prone, manual efforts [Til98]. 
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by [SLB+99, WS99, LBSOl] employed all three support meehanism eategories identified 
by [Til98], and eoneluded that re-engineering using the eombination of reverse engineering 

-2 AC 

(e.g., extraeting the most useful information) with forward engineering teehniques ad¬ 
dressed by [SLBOl] using eomputer-aided prototyping techniques (CAPS) were cost- 
effective methods for re-engineering legacy M&S software. Recent NPS research by 
[You02c] and [Pru03] support both the architectural mining and reengineering activities, 
and the follow-on simulation software application efforts. 

In addition, software evolution initiatives using prototype languages such as 
the Prototype System Description Language (PSDL) introduced by [Luq89, Luq90] and the 
Computer Aided Prototyping System (CAPS) development environment [LK88, BL94, 
Luq92, Luq94, Luq95, Luq98] support the rapid, accurate, and timely development of pro¬ 
totypes. Automated tools [WK98, LBSOl, Fav02] provide a capability to invent, correct 
and refine the conceptual models for new system architectures. This research expands 
upon the previous NPS work in reverse-engineering and forward-engineering using an en¬ 
tire Department M&S domain model as its focus. 

Different reverse-engineering tasks [Til98, YDOl, YK02] include program 
analysis, syntactic pattern matching in the programming language domain; plan recogni¬ 
tion, semantic pattern matching [SR02] in the programming language domain; concept as¬ 
signment semantic pattern matching in the application domain; redocumentation^^^, aggre- 

^07 

gation, rejuvenation and reconfiguration of assets, and architecture recovery . [Til98, 
BBW+99] also cautions that reverse engineering and the associated terminology, tools 
[DFP-l-02], and techniques are still inexact and relatively immature. 

Software engineers addressed program understanding in several cognitive 

-2 AO 

models [SFM97]. [Til98] explains two common approaches to program understanding, 
based on the software engineer’s level of domain expertise: the functionally based bottom- 
up deductive approach focused on the cognition of the implementation domain and what 
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Forward engineering is the set of engineering activities that consume the products and artifacts derived from the legacy software and 
new requirements to produce a new target system [01s95]. 

Redocumentation is the process of retroactively providing documentation for an existing system [Til98]. 
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Architecture recovery or structural redocumentation is a term for using reverse engineering to reconstruct the architectural aspects of 
software [Til98]. 

A cognitive model describes the cognitive processes and knowledge structures used to form a mental presentation of the program 
being studied [SFM97]. 
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the system does; and the top down, inductive behavioral approach, emphasizing how the 
system works, based on an existing notion of the system functionality and application do¬ 
main and employing a goal-driven method of hypothesis postulation and refinement on ex¬ 
pected artifacts. Case studies according to [Til98], show that in practice, software engi¬ 
neers switch between these two models employing a third model, the opportunistic ap¬ 
proach, depending on the problem at hand. Knowledge management techniques [SG96b, 
KNR+01, KMS02, WJC02, RJ02, Lie02, RL02, Wel02] provide possible approaches for 
reusing software engineering-related knowledge and program understanding. 

[Til98] also forwards the idea that reverse engineering as the predominant 
support mechanism used to support program understanding is an activity, which does not 
change the subject system, since it is an examination process vice an alteration process op¬ 
timally employed to identify artifacts, discover relationships, and generate abstractions. 

The process is dependent on several variables including cognitive ability and preferences, 
domain familiarity and supporting facilities to understand the three categories of artifacts 
identified by [Til98]; data, knowledge, and information^®^. 

Three canonical reverse-engineering activities support the manipulation of 
these artifacts: data gathering [Til98], knowledge management, and information explora¬ 
tion, including navigation, analysis cited by [You89]. [BM99 and Moo02] view reverse¬ 
engineering methods as a necessary step before starting software engineering activities for 
an improved system. [TK02] believe a correctly implemented reverse engineering process 
may directly benefit follow-on reengineering activities. 

2, Product Development of the Software Product Line Architecture 

a. Reengineering Architecture Assets for Software Product Lines 

Software systems have become larger, more complex, more costly and 
longer lived, which challenges the early software life-cycle models that modeled systems 
maintained for a short time, until they were retired or replaced. The software engineering 
challenge cited by [WNS+97] is how to move a large body of legacy code from its current 

309 

Data is the factual information used as the basis for study, reasoning or discussion. Knowledge is the sum of what is known, which 
includes data and information such as relationships and rules progressively derived from the data. Information is contextually and selec¬ 
tively communicated knowledge. [Til98] 
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state to a condition in which it can evolve in a disciplined way. Existing software systems 
have several inherent problems, which adversely affects maintenance [OS93]; 

-5 1 A 

• Legacy systems are usually complex, unstructured, highly coupled , with low co¬ 
hesion^", 

• Maintenance can create unpredictable ripple effects, 

• Documentation is often missing, incomplete, outdated, or unreliable, 

• The system is obsolete with unsupported hardware or software components, 

• Experienced software engineers, programmers, and software maintainers are hard to 
keep, 

• Maintenance backlogs continue to grow [OS93]. 

[OS93] defines reengineering as the bridge to an organization’s newly de¬ 
fined processes and environment from the legacy software system. Reengineering is often 
a better option than redeveloping the system when the following factors are considered: 

• Knowledge is imbedded in the software logic, 

• Reengineering allows an organization to recoup its investment of time, money and 
knowledge, 

• Legacy software is a valuable organization asset and reengineering extends the life 
of these systems. 

• Reengineered / reused code costs less than redeveloped code [OS93]. 

Eigure 10-11 illustrates the selected software architecture reconstruction ap¬ 
proach. Reengineering has the potential to improve an organization’s understanding of the 
software, establish conditions to improve future versions [01s93, 01s95, BNS97, WBS+97, 
WNS+97, BSW99, BSW+99, HDK+OO, YDOl] and better understand past failures 
[BST-l-99]. In many existing software systems, the projects were conceived, designed, and 
developed as unique products, with minimal integration, and very little systematic reuse of 
assets, which [WBS+97] suggest lose value over time by getting stale and requiring more 
assets to maintain them. 

Replacing these systems and the significant investment they represent with 
new systems, requiring a new major investment is unlikely, notes [WBS+97] while con¬ 
tinuing a fine-grained maintenance strategy, expecting the systems to evolve into maintain¬ 
able assets, appears overly optimistic. In addition to the traditional software maintenance 
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Coupling is the measure of interconnection among modules in a program structure. The lowest possible coupling is desired [Pre97]. 
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A cohesive module ideally performs a single task within a software procedure, requiring little interactions with other parts of the 
program. High cohesion is desired [Pre97] 
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activities, an assessment of the legaey system may suggest a replacement strategy, a trans- 
formation strategy, either white-box or black box , or a combination strategy 
[WNS+97]. 

[WNS+97] suggest that time-to-market and widespread distribution teeh- 
nology employing the Web and browsers will drive large-grained use of legacy system 
components integrated by middleware and object-oriented teehnologies, as opposed to 
starting development from seratch. Furthermore, [WBS+97] note that eeonomic realities 
are making low-level maintenanee aetivities unattraetive when compared with the potential 
gains from large-scale reuse, supported by a focus on architecture and produet lines em¬ 
ploying middleware and wrapping technologies. However, [BST+99], caution that several 
organizational, management, and teehnieal pitfalls may befall a reengineering projeet: 

• A flawed or incomplete reengineering strategy, 

• Inappropriate use of outside consultants and outside contractors, 

• Work force inadequately trained or tied to old technologies, 

• Legacy system not under control, 

• Requirements elieitation and validation inadequate, 

• Software architeeture is not a primary reengineering consideration, 

• No notion of a separate, discrete reengineering proeess, 

• Inadequate planning or diseipline to execute the plan, 

• Management lacks long-term eommitment, 

• Management predetermines technical decisions [BST-l-99]. 

In recognition that modern organizations own or use a significant software 
portfolio, representing a major investment of organization resources, the challenge high¬ 
lighted by [BSW+99] is to appreeiate the value of the organization’s software portfolio, 
and reduce the liability of software asset depreciation over time. The emerging emphasis 
on software architeetures and evolution of software systems provides an impetus to the de¬ 
velopment of product lines, since systems with interoperable arehiteetures allow the soft¬ 
ware to operate across defined interfaces, although [BSW+99] notes that legacy systems 
must be updated to allow them to interact as well-behaved components. 
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White-box transformation includes program understanding consisting of activities to recover lost structure or documentation 
[WNS+97]. 
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Black-box transformation includes wrapping or encapsulating the legacy system based on an understanding of the external interfaces, 
without trying to understand the internal structure [WNS+97]. 
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Figure 10-11. Software Arehiteeture Reeonstruetion 

b. Reengineering for Software Product Line Architecture Variability 

Reeall from Chapter IX that the five BMDS missile domain System-Level 
M&S possessed two hundred and eighty-five major simulation model representations, with 
only a sixty pereent availability of authoritative model representations. Note, this statistie 
only aecounts for the specific major system model representations detailed in Tables 9-1 
through Table 9-14, and does not include other model representations resident in the five 
simulations, nor does it factor in the different geodetic environments. 

It is significant to note that the lack of authoritative model representations is 
not apparent from a single simulation viewpoint. However, when viewed as part of a do¬ 
main hierarchy, against the same standard for an authoritative model representation, both 
the shortfalls and the patterns identified in Table 9-1 through Table 9-14 in Chapter IX be¬ 
come readily apparent. These shortfalls and patterns may contribute to interoperability 
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anomalies discussed earlier. This condition is the current “as-is” state of the five BMDS 
missile domain System-Level M&S. 

The scope of the future BMDS mission is unprecedented. The illustration in 
Figure 10-12 notionally represents the future “to-be” requirements for the BMDS missile 
domain System-Level M&S, with eleven possible dimensions the architectural approach 
will support. The future requirements for the BMDS missile domain System-Level M&S 
may add dimensions and significant complexity including possible variant, version, feature, 
quality, and option dimensions of future systems, in addition to the existing four dimen¬ 
sions of abstraction (e.g., structural, functional, temporal, and qualitative). The future 
BMDS requirements also add two more dimensions for the BMDS System-Level M&S, the 
cyclic life cycle support dimension and the quantitative dimension. The cyclic life cycle 
dimension support for all BMDS Element- and System-Level organizations occurs concur¬ 
rently during all three life cycle phases (e.g., research and development, transition, and op¬ 
erational support). The quantitative dimension suggests the need for quantitative meas¬ 
urements to augment qualitative identifiers, when possible, for model representation devel¬ 
opment. 

Existing BMDS element systems model representations must closely mirror 
the existing system and keep current with new capabilities, normally introduced through 
component software upgrades, suggesting a close relation between the many element sys¬ 
tem / component software requirement specifications and the component model software 
build schedules. The current BMDS model representations may be adequate for some fu¬ 
ture requirements. However, as the number of BMD system level components continue to 
grow and different manufacturers, or in the future, different countries, provide different ca¬ 
pabilities (e.g., radars, boosters, or kill vehicles), the BMDS requires the capability to as¬ 
sess the contribution of that component to the overall BMD system level capability. 

Although temporal and life cycle requirements for these future systems are 
still unknown, the BMDS may require one or more model representations for the six blocks 
from Block 04 through Block 14 to support temporal dimension requirements. Based on 
the current population of two hundred and eighty-five model representations, for planning 
purposes one model representation developed for each future block period would require 
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the near-term development of approximately one thousand model representations. The 
blue right-to-left vertieal line graphies annotated with element names and cited in the leg¬ 
end, represent this population in Figure 10-12. 



Life Cycle M&S Support 


Notional Architecture Of Missile Defense Systems Model Representation Requirements 


Figure 10-12. Future BMDS Requirements (From [GreOSb]) 

The BMDS life cycle support dimension is currently undetermined and un¬ 
der study. Recall that the Agency built the current five simulations under study to primar¬ 
ily support only the missile defense research and development phase. The emerging 
BMDS will also support the transition and operational support phases including new func¬ 
tional requirements for training, logistics, mission planning, system maintenance, and op¬ 
erational planning. Assuming for plaiming purposes these new functions generate an addi¬ 
tional single model representation requirement per simulation per block, an additional one 
thousand representations may be required. The overall requirement for future BMDS 
model representations, based on these assumptions indicates a potential need for future 
BMDS model representations numbering in the thousands. 
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The BMDS system engineering funetions requires flexible, responsive System- 
Level M&S with the ability to provide here-to-for unknown future variants, versions, fea¬ 
tures, and option requirements for spiral development within the blocks, analysis of alterna¬ 
tives for new systems, and trade space analysis for future systems or new capabilities. The 
illustration in Figure 10-12 identifies these notional top-level requirements. 

The past performance of the Department’s and the Agency’s development 
processes, practices, and techniques cited in Chapters 11 through Chapter V proved less 
than satisfactory for previous model representation requirements, and it is questionable 
whether exhortations for “better, faster, cheaper” development and V&V practices, based 
on the current practices will be sufficient to meet projected requirements cited above. The 
research strongly suggest the previous method of individual system-based development and 
low-levels of reuse, may not support optimal development methodologies for the future 
BMDS. In this effort we chose the domain management level as the optimal level to exe¬ 
cute such a program, since multiple Department enterprise efforts (e.g., JWARS, JSIMS, 
JMASS) were less than successful, and management efforts below the domain level may 
lack the resources and perspectives essential to the task 

The selected approach involves the introduction of an additional level of ab¬ 
straction beyond the current software- or component-level into a software architecture 
framework allowing the visibility of assets and the ability to leverage feature^ commonal¬ 
ity. The definition of a feature presented by [BosOO] emphasizes the early inclusion of 
quality attributes discussed in Chapter VI. Features, variants, versions, options, and quality 
attributes, with associated profiles , will be stored in the Simulation Software Architec¬ 
ture-Based Product Line Model Domain Metadata Repository illustrated in Figure 10-13. 

The Simulation Software Architecture-Based Product Line Model Domain 
Metadata Repository maintains the feature, variant, version, option, and quality attributes at 
the software product line architecture level, and supports management of the product fami¬ 
lies (e.g. weapons, sensors), software product lines (e.g. missiles, radars), and software 
product line systems (e.g. PAC-3, THAAD missiles) at the domain level with a top-down 
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[BosOO] defines a feature as a logical unit of behavior that is specified by a set of functional and quality requirements. 
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A profile is a set of scenarios that may be used as the basis for specifying a number of quality attributes such as performance or reli¬ 
ability [BosOO], 
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management approaeh, and deeentralized exeeution supporting the eoneept of software ar- 
ehiteeture-based development. 


E. THE SIMULATION SOFTWARE ARCHITECTURE-BASED PRODUCT 
LINE MODEL DOMAIN METADATA REPOSITORY 

1, The Simulation Software Architecture-Based Product Line Model Do¬ 
main Metadata Repository 

The Simulation Software Arehiteeture-Based Product Line Model Domain Meta¬ 
data Repository in Figure 10-13 provides a public interface and a defined set of services 
with layer usage dependent on lower layers. The SSA model permits a few exceptions for 
unique applications. The Simulation Software Architecture-Based Product Line Model 
Domain Metadata Repository (e.g., Metadata Repository) supports all four layers of the 
SSA Model. 



ARCHITECTURE 


-BASED 
PRODUCT 
LINE MODEL 



Layer 0 - The Real World 


Figure 10-13. SSA-Based Product Line Model Domain Metadata Repository 
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The primary purpose for the Metadata Repository is to address the root eause of 
heterogeneous system representation and data anomalies explained in Chapter IV. Major 
eomponents of the Simulation Software Arehitecture-Based Product Line Model Domain 
Metadata Repository tailored from [IS079c] provides the interfaces shown in Figure 10-14 
and includes: 

• A Domain Metadata Registry holding information describing the structure, format 
and definition of data [Ste03b]. XML recently emerged as the industry 
data/metadata interchange format of choice [DavOl] and mandated by the Depart¬ 
ment’s Joint Technical Architecture [JTA02a, JTA02b] for domain and application 
uses of specific markup languages defined by tagged data items [ASC02]. [ASC02] 
also identifies the requirements for a Department-level XML Registry (DXR) 
[Ste03a] and Clearinghouse supporting the development of a Federal-level XML 
Registry (FXR) [Ste03b, XWG02] with components developed with the appropriate 
XML Namespace [Sal02]. The Domain Metadata Registry will support the follow¬ 
ing emerging XML standards: XML query (XQuery) data models, algebra, and 
query language [CFM-l-03]; the XQuery 1.0 and XPath 2.0 Data Model [FMM-l-03]; 
and XQuery 1.0 and XPath 2.0 Formal Semantics [FMM-l-03], 

• A Domain Metadata Catalog containing instances of metadata associated with do¬ 
main data resources, supporting the use of search portals and queries exploring the 
Domain Metadata Catalog to locate relevant data (e.g., data dictionary), 

• A Domain MetaClass Catalog, which supports the development of metamodels 
with a class whose instances are classes, 

• A MetaLanguage Catalog which specifies some or all of the aspects of a MetaLan- 
guage used in the Domain (e.g., Backus-Naur form, UML, ADL, XML), 

• A Domain Meta-Metamodel Catalog or model that defines the language for ex¬ 
pressing a metamodel, 

• A Domain MetaModef^^ Catalog defining the language for expressing a mode, 

• Domain MetaObject Catalog includes all metaentities in a metamodeling language 
including metatypes, metaclasses, metaattributes, and metaassociations [DB03], 

• Domain Ontologies Catalog including data categorization schemes, thesauruses, 
glossaries, key-word lists, and taxonomies, supporting semantic and syntactic un¬ 
derstanding heterogeneous system representation data, 

• Domain Schemas Catalog representing database tables and relationship struc¬ 
tures, XML document type definitions (DTD), and XML schema, 

• Features, versions, variants, options, and quality attributes, with associated pro¬ 
files, 

• Architecture Readiness Levels (ARLs) discussed later in the chapter. 


A metamodel is a model that describes other models [IS079c]. 
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Schemas include diagrams, outlines, or models representing data structure, organization, format, structure, relationship, or data. 
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Figure 10-14. BMDS Domain Metadata Repository (DMR) 

2, The Simulation Software Architecture-Based Product Line Model Do¬ 
main Metadata Repository Registry 

The Simulation Software Architeeture-Based Product Line Model Domain Meta- 

T1 8 

data Repository Registry (e.g., Domain MRR or DMRR) structure in Figure 10-15 uses a 
conceptual data model (e.g., registry metamodel) to maintain information about data ele¬ 
ments and associated concepts (e.g., conceptual domains, value domains), including the 
metadata items described above. The DMRR maintains instances of domain metadata 
items, which define types of application level data (e.g., in a relational database schema), 
subsequently populated with real world data. A Unified Modeling Language (UML) subset 


A metadata registry maintains an information system for registering metadata [IS079c]. 
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A conceptual data model represents an abstract view of the world. A data model is a graphical and/or lexical representation of data, 

specifying their properties, structure and inter-relationships [IS079c]. 
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Metadata items are an instance of a metadata object [IS079c] 
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from [IS079c] in Appendix J. specifies the registry metamodel^^^ including metamodel 
constructs for classes, relationships, association classes, attributes , composite attrib- 
utes , and composite data types . [IS079c] uses the term “metamodel construct” for the 
model construct it uses, and the term “metadata objects” for the model constructs it speci¬ 
fies. 



Layer 0 - The Real World 


Figure 10-15. BMDS Domain Metadata Repository Registry (DMRR) 


The DMRR metamodel specifies types of classes, attributes, and relationships. A 
specific of classes, attributes, or relationship instance will be a specific type, and at any 
point in time will contain a specific value. The DMRR metamodel will be extensible, with 
the ability to add future classes, relationships, and attributes to the conceptual data model. 


A metamodel specifying a metadata registry [IS079c]. 
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A unit of measuring for modeling [IS079c]. 
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A characteristic of an object or class [IS079c]. 
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An attribute whose datatype is non-atomic [IS079c]. 
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A datatype that is also a class [IS079c]. 
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The DMRR provides a unified view of concepts, terms, value domains, and value 
meanings^^^; promotes a unifying view of the data holdings; and enables reuse and data 
sharing by providing; 

• A means for coordinating data requirements between customers and systems that 
store, exchange, or manipulate data, 

'I'yn 

• Assistance to registrars maintaining consistency among different registries, 

• A means to store, manipulate, and exchange metadata in support of data attribution, 
classification, definition, naming, identification, and registration, 

• A consistent content supporting interoperability, 

• Schema mappings of each tool set, 

• Support to translating constructs into the different languages, 

• Preservation of the concepts maintained in the original model, 

• A conceptual model to base the development of specific logical model (e.g., model 
of the information system) in an information system (e.g., database design) for the 
required application [lS079c]. 

F, ARCHITECTURE READINESS LEVELS (ARL) 

A persistent systemic problem facing decision-makers is the amount of confidence 
to place on simulation results. Research suggests this issue has many dimensions, as noted 
in the previous chapters. One of the issues involves the credibility of the model representa¬ 
tion or simulation component. Software reuse and component-based strategies consistently 
faced challenges establishing the credibility of the software component (e.g., predictable 
composition). 

Composable solutions necessitate the need for defined quality properties and a basis 
for predicting the quality properties. A major challenge of the current software engineering 
process is applying software quality models and developing reliable software productivity 
metrics [IEE92] during the early design and analysis phases of the project development. 
Chapter Vlff provided a discussion of software architecture quality analysis methods. In¬ 
terfaces pose special concerns for future component-based strategies [BBB+00]. 

Similar problem in the technology, manufacturing, and integration sectors resulted 
in process models devised as systematic metric and measurement systems, employed to 
compare the maturity between different types of technology [ACH+02]. Three Readiness 


The meaning or semantic content of a value [IS079c]. 
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A representative of the Registration Authority, which is the organization responsible for maintaining a register [IS079c}. 
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Level models illustrated in Figure 10-16 support the assessment of teehnology maturity, 
produetion readiness maturity, and integration readiness maturity for a speeifie teehnology, 
product or process technologies: Technology Readiness Levels (TRLs) [Man95], Engineer¬ 
ing Manufacturing Readiness Levels (EMRLs) [EioOl], and Integration Readiness Eevel 
(IREs) [MDA02z]. NASA space technology planning employed Technology Readiness 
Eevels (TRLs) for many years [Man95]. 


TRL 

TECHNOLOGY READINESS 

LEVELS (TRLs) (Man95] 

EMRL 

ENGR/MFG READINESS 

LEVELS (EMRLs) [FioOl] 

SYSTEM / COMPONENT / ITEM (sci) 

IRL 

INTEGRATION READINESS 

LEVELS (IRLs) (MDA02Z| 

9 

ACTUAL SYSTEM ‘FLIGHT 
PROVEN’ IN SUCESSFUL 
MISSION OPERATIONS 


TECHNOLOGIES 

MUST BE 
MATURED TO 
AT LEAST TRL 4 
OR 5 BEFORE IT 
IS READY FOR 
PRODUCTION 

9 

TECHNOLOGY PROVEN IN 

SYSTEM OPERATIONAL TEST 
/ INTEGRATED/PROVEN 

OO 

ACTUAL SYSTEM COMPLETE 
AND FLIGHT QUALIFIED IN 
TEST AND DEMONSTRATION 


8 

ACTUAL SYSTEM COMPLETE 
& QUALIFIED BY TEST AND 
DEMONSTRATION 

7 

SYSTEM PROTOTYPE 

DEMONSTRATED IN OPER- 

ATIONL ENVIRONMENT 


7 

INTEGRATION-READY TECH 

DEMO-COMP/ELEMENT 

INTEGRATED AND DEMO 

6 

SYSTEM/SUBSYSTEM MODEL 
OR PROTOTYPE DEMO IN 

RELEVANT ENVIRONMENT 


6 

TECHNOLOGY ADAPTION 
COMPONENT F3 DEMO IN 

SYSTEM ARCHITECTURE 

5 

COMPONENT/BREADBOARD 

VALIDATED IN RELEVANT 

ENVIRONMENT 

5 

IDENTICAL SYSTEM / 

COMP / ITEM 

PRODUCEDOR S/C/I PROD. 

B 

SYSTEMS ENGINEERING AND 

ANALYSIS-FUNCTIONS VAL 

BY PROTOTYPE 

4 

COMPONENT/BREADBOARD 

VALIDATED IN LAB 

4 

SYSTEM / COMP / ITEM IN 
PRODUCTION OR LRIP. 

READY FOR FULL PROD 

B 

COMPONENT F3 PROVEN 

FEASIBLE IN LAB 

3 

ANAL / EXPERIMENTAL 

CRITICAL FUNC/CHAR PROOF 
OF CONCEPT 

3 

SYSTEM / COMP / ITEM IN 

ADV DEV. READY FOR 

LOW RATE PRODUCTION 

B 

SYSTEM APLLICATION AND 

INITIAL INTERFACE DOCS 
DEVELOPED 

2 

TECHNOLOGY CONCEPT OR 

APPLICATION FORMULATED 

2 

SYSTEM IN PRTOTYPE DEMC 

BEYOND BRASS BOARD DEV 

2 

INTERFACE FORM/FIT/ 

FUNCTION (F3) DEVELOPED 

1 

BASIC PRINCIPLES OBSERVED 

AND REPORTED 

1 

SYSTEMVALIDATED IN LAB 

OR EARLY DEVELOPMENT 

1 

BASIC APPLICATION TO 

SYSTEM OBSERVED AND 
REPORTED_ 


Eigure 10-16. Current Readiness Eevel Methods 

A major element of the M&S product line architecture approach introduced in this 
research is the Architecture Readiness Eevel (ARL) process, the simulation software archi¬ 
tecture equivalent to the TRL, EMRL, and IRL processes. The ARL has three major objec¬ 
tives: 1) measure software quality early in the development process as part of the verifica¬ 
tion process, 2) identify and measure M&S quality attributes as part of the V&V process, 
and 3) identify and measure the quality of data quality attributes supporting the validation 
process. The ARL will be employed in all life cycle phases of the simulation product line 
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development and employment ineluding: 1) mining, 2) reverse engineering, 3) reengineer¬ 
ing, 4) development, and 5) W&A. The ARL supports: 

• Model representation quality attributes, 

• Simulation quality attributes, 

• Data quality attributes 

• The Department’s W&A proeess. 

The proposed ARL for improving authoritative system representations incorporates a stan¬ 
dard reference for evaluating model representations at any level of abstraction. 


ARL 

LC 

Architecture Readiness Level Summary 

9 

OP 

Verification and Validation completed successfully. 

Product Operationally Integrated in Final Version. 

8 

SL¬ 

OP 

System Implementation Complete, Validated data 

Available, Implementation Validated 

7 

SE 

Components and Interfaces Composed. Integration Testing 

Completed, Components and Interfaces meet verified specs. 

6 

SE 

System / Interface Analysis and Design. Design Specs Verified 

Against Conceptual Model, Validation Data Confirmed. 

5 

SE- 

SA 

Conceptual Model (CM) Developed, Verification and Validation 

Plan Developed, Data requirements determined. CM Validated 

4 

SA 

System Views Developed, Implementation Layer established. 

Product Determined, Metadata employed from Registry. 

3 

SA 

Component Views Developed, Component Layer established. 

Product Line Determined. Metadata registered. 

2 

SA 

Conceptual Views Developed, Conceptual Layer established. 

Product Family Determined, Metadata analyzed. 

1 

SA- 

RW 

Viewpoints / Views Developed from Real World / Core Assets, 

Referent Layer established. Metadata needs identified. 


Table 10-1. Architecture Readiness Level Summary 
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Recall our earlier discussion about the different viewpoints and views of a house 
envisioned by the plumber, electrician, and other craftsmen contributing to the construc¬ 
tion. Although they may have a primary viewpoints and views of the housed based on their 
craft, they share several constraints such as the architectural drawings; local-, state-, and 
national-level building codes; the cost and availability of certain materials; and the avail¬ 
ability, experience level, and wage requirements of helpers. 

To ensure compliance with applicable building codes, the building contractor takes 
out a permit and schedules several visits by the appropriate building inspectors during the 
construction. During the interior “roughing in” inspection the building inspector will re¬ 
view and test the work of the electrician, plumber, and HVAC specialist before the drywall 
is installed to ensure building code compliance by the different contractors. In many cases 
the building codes evolved after repetitive tragic events caused injuries, deaths, or the loss 
of property. The proposed ARL for improving authoritative system representations im¬ 
poses similar constraints and incorporates a standard reference (e.g., building code) for 
evaluating model representations. 

In Table 10-1 we present the Architecture Readiness Level Summary with nine 
ARL levels numbered one (e.g., immature) through nine (e.g., mature), very similar to the 
base TRL Model. In addition, the life cycle view of the representation matures from the 
real-world view (RW), through the four layers of the simulation software architecture (SA) 
process, into the simulation software engineering (SE) process, and finally into the opera¬ 
tion (OP) and maintenance world. Three ARL transition points support the ARL model 
maturity process at ARL layer 1 (RW to SA), ARL layer 5 (SA to SE), and at ARL layer 8 
(SE to OP). Note that in the very similar EMRL Model, technologies need to meet a mini¬ 
mum of EMRL Pour or EMRL Live before it is ready for production. The ARL Model 
takes a very similar tack for the development of authoritative representations. 

The next section provides a synopsis of each ARL, including the activities or capa¬ 
bilities associated with the specific ARL: 

ARL 1 - The lowest level of the architecture readiness level and transition point 
from the real world to the Simulation Software Architecture (SSA) referent layer. SSA 
Referent layer viewpoints from the real world lack definition and understanding. Views 
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and requirements are immature. Attribute^^^ analysis initiated. Core asset analysis is in- 
eomplete. Arehitectural eomponents establishing the view require significant descriptive 
and classification information. Metadata analysis is pending. Architecture is immature. 
Quality requirements and measurement process initiated. Validation data require¬ 
ments are unknown. The SSA is very immature and very high risk. 

ARL 2 - The second level of the architecture readiness level, the SSA Conceptual 
Layer is established. Conceptual viewpoints from the real world have limited definition 
and understanding. Views and requirements more mature. Architectural components es¬ 
tablishing the view require additional descriptive and classification information. Metadata 
analysis is largely incomplete. Product family analysis initiated. Validation data analysis 
initiated. Metrics framework and software quality metrics process initiated. Core as¬ 
set analysis is complete. Initial query sent to Domain or Enterprise Metadata Registry for 
possible item availability and reuse. The SSA remains immature and high risk. 

ARL 3 - The third level of the architecture readiness level, the SSA Component 
Layer is established. Component viewpoints from the Domain or Enterprise Metadata Reg- 

999 

istry or the SSA Conceptual Layer have more definition and understanding. Metaclass 
established. Views and requirements validated. Architectural components establishing the 
view have adequate descriptive and classification information. Metadata analysis is largely 
complete and metadata registered. Product line analysis initiated. Product metrics estab¬ 
lished. Validation data analysis is complete. Metrics framework and software quality met¬ 
rics process in progress. Core asset utilization options established. Results received from 
Domain or Enterprise Metadata Repository search for possible item availability and reuse. 
The SSA is maturing and mitigation actions developed to address the risk(s). 


A measurable physical or abstract property of an entity [IEE98d]. 
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A requirement that a software attribute be present in software to satisfy a contract, standard, specification, or other formally imposed 
document [lEE98d]. 

330 

The act or process of assigning a number or category to an entity to describe an attribute of that entity. A figure, extent, or amount 

obtained by measuring [lEE98d]. 

331 

A decision aid for organizing, selecting, communicating, and evaluating the required quality attributes for a software system. A 

hierachical breakdown of quality factors, quality subfactors, and metrics for a software system. 

332 

A function whose inputs are software data and whose output is a single numerical value that can be interpreted as the degree to which 

software possess a given attribute that affects its quality [lEE98d]. 

333 

A class whose instances are classes. Metaclasses are normally used to build metamodels [UML99]. 

334 

A metric used to measure the characteristics of any intermediate or final product of the software development process [IEE98d]. 
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ARL 4 - The fourth level of the arehiteeture readiness level, the SSA Implementa¬ 
tion Layer is established. System viewpoints from the Domain or Enterprise Metadata 
Registry or the SSA Component Layer defined and understood. Views and requirements 
are mature. Arehiteetural eomponents establishing the view have validated deseriptive and 
elassifieation information. Metadata analysis is largely eomplete and metadata employed 
from the Metadata Registry. Produet line analysis is underway. Validation data availabil¬ 
ity determined. Metries framework and software quality metries proeess determined. Core 
assets determined. Final results reeeived from Domain or Enterprise Metadata Repository. 
The SSA is mature enough to start software engineering planning efforts. 

ARL 5 - The fifth level of the arehiteeture readiness level is the transition point 
from the SSA to eoneeptual model development in the software engineering proeess. Ar¬ 
ehiteetural assets are available from the Domain or Enterprise Metadata Registry. Produet 
line eoneeptual model development initiated. Validation data availability eonfirmed. Met- 
ries framework and software quality metries transitioned to the eoneeptual framework. 
Coneeptual Model Verifieation and Validation Plan developed. Coneeptual model devel¬ 
oped. Coneeptual model validated. The arehiteeture should be matured to at least ARL 5 
before it is ready for software engineering. 

ARL 6 - The sixth level of the arehiteeture readiness level is the system analysis 
and design phase of the software engineering proeess. Interfaee eontrol doeuments are 
eomplete. Arehiteetural assets identified from the Domain or Enterprise Metadata Registry 
for eomponent or system development. Produet line eoneeptual model validated. Vali¬ 
dated data for validation phase is available. Metries framework and software quality met- 
ries transitioned to the development phase, and metries validated . Implementation Veri- 
fieation and Validation Plan developed. Analysis and design aetivities eompleted and veri¬ 
fied. 

ARL 7 - The seventh level of the arehiteeture readiness level is the eomponent and 
interfaee development phase of the software engineering proeess. ARL 7 is optional if 
eomponent-based development methods not employed. Interfaee eontrol doeuments are 
eomplete. Arehiteetural assets are available from the Domain or Enterprise Metadata Reg- 

335 

The act or process of ensuring that a metric reliably predicts or assess a quality factor [IEE98d]. 
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istry for component development. Product line eomponent model validated. Validated 
data used for eomponent validation phase. Component-level predietive metrics^^^ estab¬ 
lished. Metries framework and software quality metrics transitioned to the component- 
level development phase. Implementation Verifieation and Validation Plan developed. 
Analysis and design activities completed and verified. Components developed and tested. 

ARL 8 - The eighth level of the architecture readiness level is the system imple¬ 
mentation phase of the software engineering proeess. System interface eontrol documents 
are complete. Arehiteetural assets are available from the Domain or Enterprise Metadata 
Registry for system development. Product line system model validated. Validated data 
used for system validation phase. System-level predictive metrics established. Metrics 
framework and software quality metries transitioned to the system-level development 
phase. Implementation Verification and Validation Plan executed. System-level develop¬ 
ment, test, and integration activities completed and verified. System transitioned form the 
software engineering phase to the operation and support phase. 

ARL 9 - The ninth level of the architecture readiness level is the system operations 
and support phase of the software life cycle. Product line system model variant main¬ 
tained. Validated data is arehived. System-level predictive metrics exercised. Metrics 
framework and software quality metries monitored and analyzed. System-level develop¬ 
ment, test, and integration documentation maintained as core assets. System transitions 
from the operation and support phase into the retirement phase. 

G. DOMAIN INTEGRATED PRODUCT DEVELOPMENT TEAMS (DIPDT) 

Implementing software product lines in the Department may be even more ehal- 
lenging since the Department aequires software system as opposed to developing the soft¬ 
ware. If eredible authoritative model representations populating eredible simulations sup¬ 
porting decision-makers confidenee in the results is the strategic end we wish to aehieve; 
and the Software Arehitecture-Based Product Line Model is the way, we suggest that the 
Domain Integrated Produet Development Team (DIPDT), employing mature Produet Line 
Practices deseribed in Chapter VIII, are the means to aecomplish this strategic objective. 

A metric applied during the development and used to predict the values of a software quality [IEE98d]. 
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Recall in earlier chapters we discussed contract complexity. In Figure 10-17, a no¬ 
tional Missile Defense Agency organizational chart suggests the challenges posed by sev¬ 
eral layers of government agencies supporting the Missile Defense Agency. Many of these 
agencies in turn have contractual agreements with one or more contractors to provide ser¬ 
vices or deliverable products. The contractors in turn may have contractual agreements 
with one or more sub-contractors. The contracts may differ in many ways such as scope, 
period of performance, deliverables, contract incentives, and other contractual clauses. The 
communication channels also vary govemment-to-government, government-to contractor, 
contractor-to-contractor, organization-to-organization, and even within the respective or¬ 
ganizations. 




Figure 10-17. MDA Contract Complexity (Notional) 

Figure 2-1 and Table 7-1 identified a consistent pattern of Department institutional 
constraints affecting M&S credibility. The reoccurring, non-technical constraints such as 
process, culture, organization, and management, in addition to the more technical con- 
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straints of software development, arehiteeture, and engineering, suggest several pre- 
eonditions require resolution before the Department may realistically plan for credible 
simulation software and improved confidence in the results. One possible means to im¬ 
prove the pre-conditions identified in the research involves the development of a Domain 
Integrated Product Development Team (DIPDT). 

The Integrated Process Team (IPT) concept is not new to the Department, and many 
principles of the IPT process support the proposed DIPDT concept. The DIPDT concept 
illustrated in Table 10-2 also integrates tailored versions of the software product line meth¬ 
odology and practice line areas, cited in earlier chapters. The DIPDT concept evolved with 
the research as it became readily apparent that no “silver bullet” approach would solve the 
Department’s systemic problems with achieving credible authoritative representations. In¬ 
stead, a Domain-Managed, de-centrally executed product team approach evolved from the 
research of the Agency System-Level Core M&S. 



DIPDT Members 

M&S 

M&S 

M&S 

M&S 

M&S 


Government/COR/Contractors 

A 

B 

C 

D 

E 

1 

Agency M&S Lead 

lA 

IB 

1C 

ID 

IE 

2 

Agency System Engineering 

2A 

2B 

2C 

2D 

2E 

3 

Agency Test & Evaluation 

3A 

3B 

3C 

3D 

3E 

4 

Agency Executing Agent / COR 

4A 

4B 

4C 

4D 

4E 

5 

Model Developer PM 

5A 

5B 

5C 

5D 

5E 

6 

Model V&V PM 

6A 

6B 

6C 

6D 

6E 

7 

Model Accreditation Agent 

7A 

7B 

1C 

7D 

7E 

8 

Model Data SME 

8A 

8B 

8C 

8D 

8E 

9 

Element A M&S Eead 

9A 

9B 

9C 

9D 

9E 

10 

Element A System 1 

lOA 

lOB 

IOC 

lOD 

lOE 

11 

Element A Sys 1 Sub-Sys 1 

llA 

IIB 

lie 

IID 

HE 

12 

Element A Sys 1 Sub-Sys 2 

12 A 

12 B 

12C 

12D 

12E 

13 

Element A Sys 1 Sub-Sys 3 

13A 

13B 

13C 

13D 

13E 
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14 

Element A Sys 2 Sub-Sys 1 

14A 

14B 

14C 

14D 

14E 

15 

Element A Sys 3 Sub-Sys 1 

15A 

15B 

15C 

15D 

15E 

16 

Element A System 2 

16A 

16B 

16C 

16D 

16E 

17 

Element A System 3 

17A 

17B 

17C 

17D 

17E 

18 

Element A System 4 

18A 

18B 

18C 

18D 

18E 

19 

Element A System 5 

19A 

19B 

19C 

19D 

19E 

20 

Element A System 5 

20A 

20B 

20C 

20D 

20E 

21 

Element B M&S Eead 

21A 

21B 

21C 

21D 

21E 

22 

Element C M&S Eead 

22A 

22B 

22C 

22D 

22E 

23 

Element D M&S Eead 

23A 

23B 

23C 

23D 

23E 

24 

Element E M&S Eead 

24A 

24B 

24C 

24D 

24E 


Table 10-2. Domain Integrated Product Development Team (DIPDT) 

Table 10-2 illustrates a subset of a notional Agency DIPDT model, showing a vir¬ 
tual organization formed around the software product line. In this case the software prod¬ 
uct lines are the five core simulations cited in Chapter IX (e.g., CAPS, MDWAR, EAD- 
SIM, EADTB, MDSE) shown as Simulation A through Simulation E on the vertical axis or 
partition. On the horizontal axis of the array a partial list of key stakeholders supporting 
the software product line, both government and contractor for the actual systems / sub¬ 
systems themselves and the model developers. The system engineer, developmental tester 
[Eic94], and operational tester play significant roles. Key DIPDT members also include 
the V&V agents and the accreditation agents. 

Note line 9 through line 15 of Table 10-2. In this view of the notional VIDPT, 
Element A is further decomposed into the system and sub-system levels with representa¬ 
tives on each simulation DIPDT in which their system or sub-system is represented. Con¬ 
tracting Officer Representatives (COR) or Contracting Officer Technical Representatives 
(COTR) play a key role on the DIPDT as matrix support for contract management support. 
The objective of the DIPDT is to develop a domain integrated product team approach based 
on the primary source level of the authoritative model representation information through 

the various levels of reoccurring, non-technical constraints to the simulation product line 
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level. The simulation product line in Table 10-2 would replace the traditional tree structure 
in Figure 10-17. We envision that as the DIPDT and the SSA concept mature, and meta¬ 
data repositories appear, a wider variety of credible and architecturally sound model repre¬ 
sentations will be available to support the future development of composable simulations. 

H. SUMMARY 

Chapter X introduced the detailed research design for the Simulation Software Ar¬ 
chitecture-Based Product Line Model including five major elements supporting the method, 
specification, design, and implementation. The chapter provided an architectural analysis 
of the Department’s existing de facto M&S software architecture and introduced the Simu¬ 
lation Software Architecture (SSA), the Simulation Software Architectural Framework 
(SSAF), the Simulation Product Lines Architecture (SPLA), the Simulation Software Ar¬ 
chitecture-Based Product Line Model Domain Metadata Repository, and Architecture 
Readiness Levels (ARL). 

The Simulation Software Architecture (SSA)-based Model is an abstract software 
architecture-based horizontal foundation supporting multiple viewpoints and views. The 

- 5 ' 3'7 

Simulation Software Architectural Framework (SSAF) is a second vertical-slice architec¬ 
tural component overlaying the SSA, which includes system and environment components. 
The Simulation Product Lines Architecture (SPLA^ provides the framework to manage the 
variability^^^ [BBOO, BosOO, DMHOl, RSCOO, MNJ+02, CBB+03], features [BosOO, 
KLD02], and differences between products and comprises the third element. 

The Simulation Software Architecture-Based Product Line Model Domain Meta¬ 
data Repository provides the structure for the Domain Metadata Registry modeled from 
[IS079c] to ensure interoperability with Department, Federal Government, and private sec¬ 
tor metadata registries. The SSA and SSAF establish the architectural foundation for the 
SPLA, which supports variability and extensibility of the architectural construct. We also 
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In UML this is called a partition or a set of related classifiers or packages at the same level of abstraction or across layers in a layered 
architecture [UML99]. 

Variability refers to decisions that will be made by a member of the development team prior to system deployment [CBB+03]. 
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adapted the arehiteetural deseription eoneeptual framework^^^ from [lEEOOB] into the SSA, 
SSAE, and SPEA models composing the Simulation Software Architecture-Based Product 
Line Model. 

Core asset development is an evolving practice. Mining existing assets, a core asset 
development option, offers an organization the potential to leverage all, or part, of its cu¬ 
mulative system investments, and represents a critical practice area supporting a software 
product line initiative. However, there are significant risks achieving success developing 
software product lines. Some reasons include the high risks due to the lack of documenta¬ 
tion, the poorly maintained state of many existing systems, and the fact that many systems, 
initially developed for different purposes, lack support for current software engineering ap¬ 
proaches. Research indicates this is generally true in the Department’s M&S domain. 

The chapter provides an overview of a reverse-engineering and reengineering envi¬ 
ronment framework to improve program understanding to support software architecture 
reconstruction architecture for model representations. Reengineering has the potential to 
improve an organization’s understanding of the software and improve it and must improve 
on past failures. In many existing software systems, the developers conceived, designed, 
and developed projects as unique products, with minimal integration, and very little sys¬ 
tematic reuse of assets, which lose value over time by getting stale and requiring more as¬ 
sets to maintain them. 

Software architecture reconstruction provides a mechanism for the effective reuse 
of software assets. Architecture reuse is the cornerstone practice of product line develop¬ 
ment, supported by three sources for architecture recovery: 

• Source code, which is authoritative, but does not hold all the information, 

• Documentation, which is normally undependable, incomplete, or outdated, 

• Human experts, who may be dependable, but biased [CW98]. 

Architecture recovery and reconstruction is an iterative and interactive labor- 
intensive effort, dependent upon the level of specification, documentation, dissemination 
and control of the legacy architecture artifacts. Architecture recovery methods vary from 
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The conceptual framework is the term of reference for the architectural description and establishes the terms and concepts for the 
content and use of architectural descriptions [lEEOOb], 


325 



entirely manual reeonstruetion proeesses to tool-supported manual reeonstruetion and semi- 
autonomous reeonstruetion teehniques, and may inelude data mining and employment of 
arehiteeture deseription languages. Pattern analysis supports architeeture recovery and re¬ 
construction efforts. 

The Architecture Readiness Levels (ARL) supports future architectural components 
and products developed from this methodology, is the fifth and final contribution. The 
chapter also suggests a complementary Domain Integrated Product Development Team 
(DIPDT) approach for implementing the Simulation Software Architecture-Based Product 
Line Model, to reduce the cycle time for resolving the multiple dimensions causing hetero¬ 
geneous system representation anomalies, and improving federation interoperability. 
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XI, CONCLUSIONS AND RECOMMENDATIONS FOR FUTURE WORK 


A. REVIEW OF MAJOR RESEARCH AREAS RELEVANT TO THIS WORK 

This research focuses on the major research areas introduced in Figure 2-1, and rep¬ 
licated in Figure 11-1 for improving the current heterogeneous system representation 
anomalies-based issues, which erodes credibility in the Department simulation process, and 
injects doubt instead of confidence in the simulation results supporting Departmental- and 
National-level decision-makers. 



Figure 11-1. Major Areas Researched for this Work 

In Chapter 11 we identified the seven major areas affecting improved credibility in 
simulation model representations supporting confidence in the Department- and National 
Level decision-making process: 

• An Architectural Framework, 

• Software Engineering, 

• Process Improvement, 
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• Quality, 

• Verification and Validation, 

• Heterogeneous Data, 

• Technical and Substantive Interoperability. 

Five of the seven areas affecting simulation credibility were methods to improve the 
simulation product, and two of the seven areas (e.g., heterogeneous data, technical and sub¬ 
stantive interoperability) were conditions adversely affecting model credibility. While all 
of the five simulation product improvement methods experienced some progress, they also 
encountered several limitations: 

• The Department’s M&S architectural framework remains immature. The Common 
Technical Framework including the High-Level Architecture, Conceptual Model of 
the Mission Space, and improved data quality initiatives did not significantly im¬ 
prove simulation model representation credibility. The High-Level Architecture 
achieved technical success, and the IEEE adopted it. However, it is too early to tell 
whether the HLA will be a commercial success, now that the Department no longer 
financially underwrites it. The Conceptual Model of the Mission Space model im¬ 
plementation proved inconsistent, reducing credibility in the V&V process. Data 
remains a major problem [DoDOla]. The current object models (e.g., SOM, EOM, 
BOM) do not address the substantive interoperability issue. In addition the JTA 
and the HEA remain incompatible. 

• The Department’s software engineering strategy to replace many large-scale legacy 
simulations with new, well-engineered, joint simulations (e.g., JMASS, JSIMS, 
JWAR) experienced significant cost overruns, failed to meet continually extended 
delivery schedules, and met only a subset of the approved requirements. With ma¬ 
jor funding reductions or planned cancellation, the future of the three major simula¬ 
tion software engineering initiatives appear limited at best. As a result the Depart¬ 
ment is reviewing options to extend the life span of the existing large-scale, legacy 
simulation systems. 

• Although process improvement in no way guarantees fewer errors in the final soft¬ 
ware product, initial indications over the past decade suggest that properly disci¬ 
plined process improvement has a favorable impact on software development qual¬ 
ity and may improve simulation model credibility. Several process methodologies 
exist including the CMM, CMMI, Software Acquisition Management, and the 
Product Eine Practice Areas. Our research indicates that mature processes are pre¬ 
conditions for the development of credible simulation model representations. 

• Quality remains an elusive element in the Department’s M&S portfolio. In many 
cases the discussion of M&S quality eventually lead to debates about fidelity. Ei- 
delity as a term has many appropriate uses, but some suggest that fidelity is an 
overused term, open to almost any meaning. Our research indicates that quality at¬ 
tributes are also pre-conditions for simulation credibility, and are key architectural 
components of the proposed Simulation Software Architecture Model. 
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• The Department’s V&V proeess is the primary method for establishing simulation 
eredibility. However, systemie issues with incomplete or non-existent conceptual 
models, a general lack of validation data, and inconsistent V&V implementation 
raises many questions of a simulation’s credibility. The accreditation process, de¬ 
pendent upon a sound V&V process, lacks credibility as a result. 

The research results presented earlier in Chapters II and IX indicates that after fifty 
years of development the Department M&S domain has not achieved the desired level of 
credibility and confidence needed to support the Department decision-making process, and 
existing methods, techniques, and procedures have not achieved the desired objectives. 

A major finding of this research suggest that there is little supporting evidence that 
the Department M&S software development community will achieve future success: 1) im¬ 
proving heterogeneous system representation anomalies, 2) improving overall simulation 
credibility; and 3) improving confidence in results provided by Department M&S employ¬ 
ing current conceptual model development paradigms. In addition, many of the past meth¬ 
ods and recommendations to improve the Department’s M&S portfolio have been reiterated 
and reissued, and it is uncertain if more time and more resources will result in: 

• Better verification and validation (V&V) techniques, 

• Better discipline in the software development process, 

• Improved accreditation processes, 

• Improved use of data, 

• Improved quality, 

• Improved documentation, 

• Improved coordination between the users and the developers, 

• Improved credibility, 

• Improved confidence in the results. 

B, EVALUATION OF A SIMULATION SOFTWARE ARCHITECTURE 
METHOD AGAINST THE CURRENT M&S TECHNIQUES 

It is clear from the research that Department and the Service Components acted 
with due diligence to improve the quality of Department M&S. By the mid-1990s the De¬ 
partment established M&S management organizations, developed policies, implemented 
plans, allocated significant funding, and executed major new programs with the goals and 
objectives of improving Department M&S practices. However, not withstanding these 
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considerable efforts, major eoneerns still exist about the eredibility of the Department 
simulation proeess. There are several major reasons for this lack of credibility in Depart¬ 
ment M&S. 

The first contributor was the software industry-wide hope for a “silver bullet” solu¬ 
tion to resolve the Department “software crisis”. However, as hope for a “silver bullet” 
teehnieal breakthrough to improve Department software, including simulation software, 
waned in the 1990s, it beeame apparent that other non-teehnical solutions were required. 

In addition, new Department M&S management organizations, sueh as the DMSO, had to 
overcome Service paroehialism, the unpredictability of the Department budget process, the 
vagaries of the Department aequisition proeess, and over forty years of unplanned Depart¬ 
ment M&S growth. Then in the late 1990s signifieant funding and manpower resources 
originally programmed to improve Department M&S were committed to the Year 2000 
problem resolution project. 

A second underlying cause for the current situation is the Department institutional 
constraints affecting M&S credibility, and the apparent slow maturity progress of major 
M&S credibility factors cited in Table 7-1. In 1996 a major Department M&S study 
[PHP+96] identified three major types of systemie Department M&S challenges, technical, 
cultural, and managerial; while eoncurrently the Simulation-Based Aequisition (SBA) ini¬ 
tiative identified three similar factors, process, culture, and environment as SBA enablers. 
More recently the SEI endorsed three related practice areas, teehnieal management, and 
organizational, to support the successful implementation of software produet lines. 

This is significant since these faetors represent an institutional realization that the 
resolution to the twenty year old “software crisis” is not a “silver bullef ’ solution. Instead, 
the problem and the solution are multi-dimensioned, complex issues that require ehanges in 
many areas. This change will take time for the Department to truly assimilate and adapt to 
during the transition phase into the Information Age. We must however, also be mindful of 
the many Industrial Age vignettes where nations, armies, navies, or air forces were slow to 
adapt and the severe consequences of that that failure in terms of defeat or loss of life. 

Software quality, a third contributing study factor, continues to lag well behind the 
eomputer industry’s hardware engineering segment progress on quality, cost, and perform- 

330 



ance. While computer hardware engineering is well into the fourth computer era identified 
by [Pre97] and employs the production approach to quality according to criteria established 
by [Wal94], simulation software quality is best labeled as transcendental. In contrast to 
the mature computer hardware-engineering sector, the software engineering sector of the 
computer industry is just now achieving a consensus on a software engineering body of 
knowledge and agreement on the benefits of software process improvement. The imple¬ 
mentation of Department M&S quality objectives for improving quality attributes, includ¬ 
ing fidelity, accuracy, resolution, error, detail, aggregation/disaggregation, and multiresolu¬ 
tion modeling techniques have achieved only limited success. 

Software architecture is the fourth study factor acknowledged to strongly influence 
a system over its life cycle. While computer hardware-related architectures supported by 
Moore’s Law were ascendant over the past fifty years, software-related architectural con¬ 
siderations, if they existed, were of secondary importance to the overall system engineering 
and development process. Today, the cost and complexity of software-intensive systems 
have changed the industry’s dynamics and the publication of software architectural descrip¬ 
tions [lEEOOb] and styles [SC96, BK99, CBB+OS] for software-intensive projects are in¬ 
troducing software architectural principles to the emerging software engineering commu¬ 
nity. However, the Department’s life cycle management of software-intensive simulation 
systems lacks software architecture concepts beyond the application of the CTF (e.g., HLA, 
Conceptual model, data). 

The Department legacy M&S portfolio is the fifth major study factor. The Depart¬ 
ment developed a significant and expensive portfolio of legacy M&S over the past fifty 
years. Although an operational system has a de facto architecture, many of these M&S 
evolved in an ad hoc manner and have become large legacy systems with multiple stake¬ 
holders and expensive support infrastructures. A significant number of these systems lack 
conceptual models and have an ad hoc V&V history. In some cases the developer created 
conceptual models after the fact, if at all, and adherence to disciplined M&S V&V process 
was a low priority. 

Department agencies also have a tendency to use these systems for studies or tests 
for which they are ill suited, in order to show a return on investment. As a result the ac- 


331 



ceptability criteria and caveats developed by the accreditation process for these simulations 
may be of limited value or counter-productive to the original purpose and intent of the 
study or test objectives. Complicating the current situation, replacement systems are over¬ 
due. The new Joint M&S programs (e.g., JWARS, JSIMS, JMASS) scheduled to replace 
many legacy systems are all late, over cost, and reducing the original requirements to meet 
a future operational status. 

M&S verification and validation, a sixth study factor, is the foundation for M&S 
credibility and confidence. However, many consider V & V expensive and time consum¬ 
ing. In this research we reviewed the V&V process in the context of supporting Depart- 
ment-and National-level decisions with improved credibility and confidence, and con¬ 
firmed previous assessments indicating the quality of Department V&V practices are ad 
hoc and inconsistently applied. 

The conceptual model of the mission space (e.g., CMMS or FDMS), a seventh 
study factor, is the approved foundation for a rigorous Department verification and valida¬ 
tion program. The Department bases successful V & V on a verified and validated concep¬ 
tual model. In practice, conceptual model development and conceptual model validation as 
a precursor to verifying the software implementation, rarely occurs. The M&S community 
developed many conceptual model formats [RPGOO], however, there is no consensus on a 
standard conceptual model at this time. 

The current Department M&S Hierarchy, the eighth study factor, provides only 
general, qualified levels for M&S fidelity and resolution, provides only limited support for 
the user to ascertain a simulation’s attributes. The model provides little if any support for 
quantifiable measures of simulation quality. In addition, metrics based on this model tend 
to be qualified, subjective, and lack authority. In this research we analyzed the current De¬ 
partment M&S Hierarchical model as a core asset and reviewed alternatives for mining re¬ 
verse engineering and reengineering these core assets into product line architecture. 

Common limitations of the Department’s efforts to improve M&S credibility and 
confidence in results include the following reasons. First, a significant amount of current 
Department M&S systems were developed in previous decades and are currently in a main- 
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tenance mode, suggest a strategy of introdueing improvements in a planned evolutionary 
manner. 

Seeond, the M&S policies, procedures, best practices, and funded projects devel¬ 
oped in the mid-1990s may need more time to promulgate and affect M&S development 
processes for new models and simulations. 

Third, in the 1990s, the Department experienced a major shift from M&S support¬ 
ing a bi-polar world with two super powers to a world characterized by asymmetrical war¬ 
fare and weapons of mass destruction, and support Department transformation initiatives, at 
a pace of change seldom experienced in the Department. 

Fourth, as the Department evolves from a threat-based, requirements-driven organi¬ 
zation and transforms itself to a capability-based model while concurrently maintaining the 
ability to execute the National Military Strategy, the question remains, are we developing 
the right M&S (e.g., are we building the right thing)? 

Last, if we determine that we are building the right thing, have the software engi¬ 
neering capabilities of the Department matured enough to “build it right?” The product 
line domain architecture supporting specified requirements allocated from the system engi¬ 
neering process presented in Chapter X of this dissertation address these limitations. 

Accomplishing this significant undertaking requires a mature population of soft¬ 
ware engineers and software architects. Grounded in the technical foundations of the field 
(e.g., computer science, engineering, information technology), and the emerging software 
architecture discipline, future software engineers and software architects will need to un¬ 
derstand the viewpoints of multiple, diverse stakeholders at all levels of the Enterprise and 
synchronize composable software engineering solutions that meet the cost, performance, 
and schedule constraints. 

Distributed, integrated product development teams software engineers and software 
architects working with authoritative product line repositories of valid domain data provide 
one such method for improving future simulation credibility. Software engineering and 
software architectures are now emerging as critical domain disciplines that must continue 
to mature the corresponding bodies of knowledge, if the Department and the Nation are to 
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exploit the full potential of computer technology, and set the foundation for the next great 
era in computing, the Software Architecture Era. 

C. RECOMMENDATIONS FOR FUTURE RESEARCH 

The simulation software architecture-based product line approach to simulation 
credibility, supported by architecture readiness levels and domain integrated product devel¬ 
opment teams provides five proposed areas for future research. The Simulation Software 
Architecture (SSA)-Based Model provides an abstract software architecture-based horizon¬ 
tal foundation supporting multiple viewpoints and views. Second, the Simulation Software 
Architectural Framework (SSAF) adds a second vertical-slice architectural component 
overlaying the SSA, which includes system and environment components. 

These architectural artifacts support the Simulation Product Fines Architecture 
(SPFA) providing the framework to manage the variability, features and differences be¬ 
tween products. The fourth component, the Simulation Software Architecture-Based Prod¬ 
uct Fine Model Domain Metadata Repository provides the structure for the Domain Meta¬ 
data Registry to ensure interoperability with Department, Federal Government, and private 
sector metadata registries. The Architecture Readiness Fevels (ARE) to measure future 
architectural components and products developed from this methodology is the fifth area 
suggested for future research. 

1. Extension of the Simulation Software Architecture (SSA)-Based Model 
into Different Domains 

The lack of a true software architecture foundation for the Department’s simulation 
system development is the equivalent of being on the road to Abilene^"^° without a plan, and 
nobody knows why we are going. The SSA Model complements and supports the existing 
HFA, with a focus on improving the conceptual models and data elements of the current 
Common Technical Framework. Figure 11-2 illustrates the current plan to implement the 
Missile Domain SSA. The first phase involves decomposing the domain M&S hierarchy. 
The second phase involves the identification of patterns for future product lines. Chapter 
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Defense Systems Management College teaching point in the Program Management Course reinforcing the need for a program plan. 
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IX provided the pattern analysis for the first two phases. The final phase involves the eore 
asset mining and arehiteeture reeonstruetion for the SSA. 

The development of views and viewpoints suggest the need for a broad-based re- 
seareh effort in multiple domains and the development of the domain metadata repository 
to support the development of simulation software arehiteeture deseription (SimSAD). 
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Figure 11-2. Missile Defense Domain SSA Status (After [GreOSb]) 

2, Simulation Software Architectural Framework 

We identified the SSAF supporting the Simulation System Arehitectural Deserip¬ 
tion (SimSAD), as the system’s environment, which includes the physical world and the 
external objects, conditions, or processes influencing the behavior of the system in past, 
present, and future dimensions, real or imagined, but clearly defined and documented in the 
SimSAD. The SimSAD provides the foundation for the follow-on SPLA to manage the 
product line variability of a SimSAD. 
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In the missile defense domain, future researeh would determine if the SimSAD pro¬ 
vided the sealability to manage the large number of simulation models projeeted to support 
a dynamie bloek evolution of the missile defense system in addition to unplanned experi¬ 
ments and exeursions. Future researeh on the environment eomponent of the SSAF could 
provide insights into the viability of this approach in supporting a major weapon system 
acquisition, which must function perfectly the first time in a worldwide environment. 

3. Simulation Product line Architecture 

The SPLA provides the architectural capability to exploit commonality. There is a 
potential requirement for thousands of authoritative simulation model representations. In 
addition, the BMDA capability-based acquisition process may add new elements, while 
terminating or reducing other element development. This scenario indicates an additional 
need to maintain accurate configuration management of variant, version, feature, and op¬ 
tion requirements of past and future unknown systems. New system developments may 
also add more complexity since the systems may be unprecedented. Additional research 
could identify the boundaries for the SPLA. 

4, SSA Domain Metadata Repository Model 

The SSA Domain Metadata Repository, when fully populated will maintain a Do¬ 
main Metadata Registry, a Domain Metadata Catalog containing instances of metadata as¬ 
sociated with domain data resources, supporting the use of search portals and queries; a 
Domain MetaClass Catalog, which supports the development of metamodels with a class 
whose instances are classes; A Metalanguage Catalog which specifies some or all of the 
aspects of a MetaLanguage used in the Domain; a Domain Meta-Metamodel Catalog or 
model that defines the language for expressing a metamodel; a Domain MetaModel Cata¬ 
log defining the language for expressing a mode; a Domain Ontologies Catalog including 
data categorization schemes, thesauruses, glossaries, key-word lists, and taxonomies; a 
Domain Schemas Catalog representing database tables and relationship structures, XML 
document type definitions (DTD), and XML schema; Features, versions, variants, options, 
and quality attributes, with associated profiles; and Architecture Readiness Levels (ARLs). 
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Additional research may evaluate if data is visible, available, and usable when needed and 
where needed to speed decision-making, including the DoD Metadata Registry identified in 
Figure 11-3. 
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Figure 11-3. Future Areas of Research (From [Ste03b]) 


5, Architecture Readiness Levels 

Architecture Readiness Levels provide the opportunity to anchor simulation credi¬ 
bility and decision-makers’ confidence in an architectural foundation with quality attrib¬ 
utes. Additional research in improving the V&V process based on the architectural matur¬ 
ity of the model representations has potential to improve decision-makers’ confidence in 
future simulation results. 
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D. CONCLUDING REMARKS 

Chapter I posed the following question; How may the Agency better define, de¬ 
velop, evolve, manage, and maintain authoritative model representations in the five se¬ 
lected large-scale, software-intensive legacy simulations and effectively address simulation 
interoperability issues (e.g., heterogeneous system representation anomalies) within exist¬ 
ing constraints and pre-conditions (e.g., technical, organizational, managerial) affecting 
credibility in the five selected simulations systems? 

The following hypothesis was posed in response to this question: Employing a do¬ 
main-managed, four-layered simulation software architecture-based product line model 
with a referent layer, developed in a distributed domain integrated software engineering 
environment supporting the evolution of five selected Agency legacy simulations can im¬ 
prove the authority of representations in the five selected missile defense simulations. The 
Software Architecture-Based Product Line Model developed in a distributed development 
environment by a Domain Integrated Product Development Team employing Architecture 
Readiness Levels (ARL) to measure quality provides such a methodology. Hypothesis test¬ 
ing included concurrent implementation of the techniques and procedures in the Agency’s 
M&S domain. The Software Architecture-Based Product Line Model for Simulation 
Model representations provides such a methodology. 
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APPENDIX A - BALLISTIC MISSILE DEFENSE ORGANIZATION (1996-2001) 
A. INTRODUCTION 

During the 1996-2001 period, the Ballistic Missile Defense Organization (BMDO) 
consisted of two major components, a Theater-based missile defense program and the Na¬ 
tion’s National Missile Defense program. Figure A-1 identifies the major U.S. missile de¬ 
fense acquisition programs existing in 2001, including international collaborative efforts, 
major technology programs, and the supporting BM/C2, weapon, sensor, and communica¬ 
tion components. Figure A-1 also identifies the service or organization(s) responsible for 
the management of the program development for each system [Bau02]. 


BMDO BMDO 



Figure A-1. Missile Defense Organization - 2001 - (After [BMDOOa]) 


B, THEATER MISSILE DEFENSE (TMD) ORGANIZATION - (1996-2001) 

No single system can possibly protect the diverse and complex threat posed by en¬ 
emy ballistic missiles, aircraft, and cruise missiles against U.S. and coalition forces, se¬ 
lected assets and population areas within a theater of operations. The TMD and supporting 
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the Joint Theater Air and Missile Defense (JTAMD) doctrine plans to detect, classify, in¬ 
tercept, and destroy/negate enemy missiles prior to launch, or while in flight. The BMDO, 
the Joint Theater Air and Missile Defense Organization (JTAMDO), Combatant Com¬ 
manders, and other Department Components developed a Family of Systems (FOS) 
[BMDOOb] concept to provide a coherent, cohesive, and effective protection capability. 

The FOS incorporated four major pillars; Attack Operations^'^', Active Defense of post- 
launched theater missiles. Passive Defense^"^^ through early warning, and an integrated Bat¬ 
tle Management, Command and Control, Communications, Computers, and Intelligence 

'lA'i. 

(BMC4I) to provide missile defense in-depth. Figure A-2 illustrates the complex enemy 
threat and the JTAMD Family of Systems developed to counter them. 


THREAT TYPES 



Figure A-2. 


Family of Systems (After [BMDOOa, BMDOOb]) 


Attack operations includes both pre- and post-launch attack of enemy transportor- erector- launchers (TELs) and supporting infra¬ 
structure [BMDOOa], 
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Passive defensive measures include actions such as deception, dispersion, or protective construction to minimize the enemy ballistic 

missile defense effectiveness [BMDOOa], 

343 

BMC41 includes the controlling and coordinating procedures and systems to integrate an effective defense [BMDOOa], 
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1 . 


PATRIOT - (1996-2001) 


The Army’s Patriot missile defense system, managed by the Army, has evolved 
sinee 1965, first as the Surfaee to Air Missile Development (SAM-D), followed by Pre- 
Planned Produet Improvements into the Patriot Anti-Taetieal Missile - 1 (PAC-1) and -2 
(PAC-2) systems. In Operation Desert Storm, the Army quiekly upgraded the PAC-2 sys¬ 
tem form an air defense system with the Quiek Response Program (QRP) and Guidanee 
Enhaneement Missile (GEM) modifieations in 1993 to eounter the new SCUD missile 
threat, emerging as the PAC-2 GEM version of the system. The eurrent PAC-3 system 
[MDA02v] employs HTK teehnology and eonstitutes the lower tier of a layered defensive 
eapability supporting the Theater Missile Defense eoneept. Major PAC-3 eomponents in- 
elude the BMC4I, sensors, and weapon system. In 2000, the system was preparing for In¬ 
dependent Operational Test and Evaluation (lOT&E) and a produetion deeision [BRC+01]. 

2, THEATER AREA AIR DEFENSE (THAAD) PROGRAM - (1996- 
2001) 

THAAD is an Army theater missile defense (TMD) system [MDA02u] designed to 
engage the full speetrum of theater elass ballistie missile threats, employing HTK teehnol- 
ogy. THAAD was designed to meet eritieal warfighters needs to intereept longer-range 
theater-elass ballistie missiles at high altitudes and further away from the intended target. 
The THAAD system is the only defensive system that is effeetive inside and outside the 
earth’s atmosphere, and may be quiekly deployed worldwide on short notiee. Major 
THAAD eomponents inelude the BMC4I, sensors, and weapon system. 

3. NAVY AREA DEFENSE (NAD) PROGRAM- (1996-2001) 

The NAD is a lower tier system designed as a taetieal missile defense system 
against endoatmospherie missile threats within the earth’s atmosphere. The Aegis weapon 
system equipped with the phased array, SPY-IB/D radar, and BMC4I was the foundation 
weapon platform for the NAD. It will evolve into the future sea-based missile defense ea¬ 
pability of the BMDS. 
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4. 


NAVY THEATER-WIDE (NTW) DEFENSE PROGRAM- (1996-2001) 


The NTW missile defense system [BMD99a] was a tactical missile defense system 
against medium-and intermediate-range theater ballistic missiles outside the earth’s atmos¬ 
phere. The NTW will have an improved interceptor, phased-array SPY-IB/D radar, and 
BMC4I. With the evolution of the modified Standard Missile (SM-3) and enhancements to 
the Aegis Weapon System, the NTW defense system has become a component of the sea- 
based midcourse segment of the BMDS. 

5. AIRBORNE LASER DEFENSE PROGRAM (ABL)- (1996-2001) 

The ABL system [MDA02o] is a rapidly deployable airborne platform: a 747-400F 
aircraft, equipped with a long-range laser weapon to acquire, track, and negate TBMS in 
the boost phase of flight. The ABL will be capable of operating above cloud-level for ex¬ 
tended periods of time, providing wide-area geographic and near 24-hour temporal cover¬ 
age of potential TBM launch areas. The ABL weapon system includes the aircraft, 

BMC4I, laser, and beam control/fire control segments. 

6. MEDIUM EXTENDED AIR DEFENSE SYSTEM (MEADS)- (1996- 

2001) 

The Medium Extended Air Defense System (MEADS) [MDA02w] is an interna¬ 
tional cooperative terminal missile defense program shared by the U.S., Germany, and Italy 
to provide 360° protection for maneuver forces against short-range tactical ballistic mis¬ 
siles, cruise missiles, and unmanned aerial vehicles. MEADS will replace HAWK and 
PATRIOT air defense systems with point and area defense capabilities. The system is 
composed of a surveillance radar, fire control radar, launcher, missile, and Tactical Opera¬ 
tions Center. 
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7. 


ARROW WEAPON SYSTEM- (1996-2001) 


The Arrow Weapon system [MDA02t], jointly developed by the U.S. and Israel, 
provides short-and medium-range ballistie missile defense of Israel and U.S. forees de¬ 
ployed in the region. 

C. NATIONAL MISSILE DEFENSE (NMD) ORGANIZATION - (1996-2001) 

The NMD will provide proteetion from a limited attack on the United States by bal¬ 
listic missiles. The NMD program [BMDOOc, BMDOOi] exploited the products of previous 
technology and development programs, including the ground-based radar (GBR) prototype, 
development of the X-band phased-array radar^"^"^, passive infrared technology from the 
Brilliant Pebbles program, and advanced tracking algorithms from the Exoatmospheric Re¬ 
entry Vehicle Interceptor System (ERIS) and High Endoatmospheric Defense Interceptor 
(HEDI) programs. The NMD system [BMDOOc, BMDOOi] is comprised of the following 
major components: 

1. NMD GROUND-BASED SENSORS- (1996-2001) 

Ground-based sensors include the Upgraded Early Warning Radars (UEWRS) 
[BMDOOh, MDA02g], based on existing Pave Paws and Ballistic Missile Early Warning 
Systems (BMEWS), and the X-Band Ground-Based Radar (GBR/XBR)^"^^ system 
[BMDOOg]. 

2. NMD SPACE-BASED SENSORS- (1996-2001) 

Space-based Sensors include the existing Defense Support Program (DSP) satellite 
system^"^^ [BMDOOh, MDA02g], the planned deployment of two future systems, the Space 
Based Infrared System (SBIRS) High [BMDOOh, MDA02g] to provide missile launch, de- 
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XBR efforts included improved software operatrions, applications processing, and new radar imaging technologies [BMDOOa], 
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The XBR phased-array system conducts track-sweeping operations electronically, which is must faster than mechanically based 
antennas. With high resolution, the short wavelength of the X-band radar supports early tracking, identification, and discrimination of 
targets some 2,400 mile away [BMDOOa]. 
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The DSP system, initially deployed in 1971, includes a constellation of satellites in geosynchronous orbit 23,000 miles above the 
earth, linked to earth stations to provide missile launch warning and limited tracking capabilities [BMDOOa], 
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tection, surveillance, and track data; and the Space Based Infrared System (SBIRS) Low 
[BMDOOh, MDA02g] to provide more discrete identification, discrimination, and tracking 
capabilities early in the missile’s flight. These systems will also provide an assessment of 
post-intercept results. SBIRS Low builds on the earlier success of the Midcourse Space 
Experiment (MSX), which demonstrated the capability of long-wave infrared surveillance 
and discrimination from space and will support both TMD and NMD systems. 

3. NMD GROUND-BASED INTERCEPTOR (GBI) - (1996-2001) 

The NMD Ground-Based Interceptor is an enhanced booster stack composed of 
available commercial-off-the-shelf booster components. The booster delivers the Exoat- 
mospheric Kill Vehicle (EKV), a kinetic energy HTK interceptor into engagement range 

'lA'i 

for a midcourse intercept of the ballistic missile warhead [BMDOOe, MDA02q]. 

Battle Management/Command, Control, and Communications (BM/C ) system is actually a 
composite of three discrete components: battle management, command and control, and 
communications. Within the NMD system these components consist of the software, 
hardware, facilities, and communications for planning, tasking, and controlling, ensuring 
positive human control over all aspects of the NMD system [BMDOOa, BMDOOf]. 
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A ballistic missile is an intercontinental class missile with a range in excess of 5,500 kms [BMDOOa], 
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APPENDIX B - BALLISTIC MISSILE DEFENSE SYSTEM (BMDS) PROGRAM 

2002-PRESENT 

A, BMD SYSTEM OBJECTIVES 

The Agency will develop the Ballistic Missile Defense System incrementally with 
layered defenses to intercept ballistic missiles of all ranges in all phases of flight—boost, 
midcourse and terminal [Rum02c] as illustrated in Figure B-1. 



Figure B-1. The BMDS Mission Scope (After [GreOSb]) 


The existing FoS capabilities will continue to evolve with the BMDS, however the 
combination of targets, defense systems, and layered defensive options illustrated in Figure 
B-1 will necessitate major changes to the BMD M&S program. The BMDS is a single sys¬ 
tem managed as an integrated program, a major programmatic shift from the MDAP- 
centric systems previously managed by the Services. The BMDS will consist of elements 
configured into layered defenses to provide autonomous and mutual support, which will 
provide multiple engagement opportunities against a threat ballistic missile in all phases of 
flight. 
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B. 


BMD SYSTEM-LEVEL CAPABILITY 


The Department provided the MDA with new guidance and direction identified in 
Table B-2 to develop the BMDS. 


Previous Direction 

New Direction 

Treaty Constrained 

Withdrew From ABM Treaty June 2002 

Requirements-Based 

Capability-Based 

Major Defense Acquisition Program 
(MDAP) Focus 

Ballistic Missile Defense System (BMDS) Focus 

Joint Requirements Oversight Council 

Senior Executive Council (SEC) And Missile Defense Support 
Group (MDSG) 

5000 Series Acquisition Life Cycle 

3 Phases: RDT&E, Transition, Procurement By Blocks 

Standard DoD Contract Authority 

Other Transaction Authority (OTA) - National Team 

R&D / Acquisition Functions 

R&D / Acquisition Functions / IDO 

Mission Needs Statement (MNS) / 
Operational / Capstone Requirements 
Documents (ORD / CRD) 

System Technical Objectives And Goals (TOG) 

Capability-Based ORD 

Developmental / Operational Test 

Developmental / Operational Test 


Table B-2. Changes to BMDS Direction 

The BMDS system engineering process [MDA021] will allocate these system capa¬ 
bility specifications cited in Table B-3 to BMDS segments/elements/systems/ subsystems 
in Table B-4 through a series of documents including the BMDS Technical Objectives and 
Goals (TOG), the Adversarial Capability Reference Document (ACRD), System Evolution 
Plan (SEP), Integrated Master Plan, Integrated Master Schedule, and System Capability 
Specification (SCS), among others. The MDA system engineering function and M&S pro¬ 
gram must be able to support the development, maintenance, and testing of the unprece¬ 
dented BMDS and must successfully address: 

• Operational realities 

• Engineering complexities 

• Transitioning from a requirements-based process to a capabilities-based approach 

• Implementing an adversarial capability approach instead of a clearly defined threat 

• Implementing an evolutionary acquisition strategy using a two-year block approach 
[MDA021]. 
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Table B-3. Required BMDS Capability 

C. BMD SYSTEM-LEVEL LAYERED DEFENSE CAPABILITY 
1. BOOST PHASE DEFENSE SEGMENT (BDS) 

The Boost Phase Defense Segment (BDS) [MDA02i, MDA02j, MDA02n] is de¬ 
signed to eounter that part of a threat missile’s flight path from launch until it stops accel¬ 
erating under its own power. At this point in its flight the threat missile has been climbing 
against the Earth’s gravity for 100 to 300 sec. and may achieve an altitude of 200 km. The 
Boost Phase Defense (BDS) intercept of the missile’s flight by the ABL [MDA02n] under 
development as a system and other future BDS is the best solution, allowing for the defense 
of a very large area of the Earth, and destruction of the target before the deployment of 
midcourse countermeasures. 


2, MIDCOURSE DEFENSE SEGMENT (MDS) 

The Midcourse Defense Segment (MDS) [MDA02i] is that part of the ballistic mis¬ 
sile trajectory where the missile has completed the thrust phase and is following a more 

predictable flight path during the mid-course phase as it freefalls towards its target. This 
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phase may last as long as 20 minutes in the ease of an intercontinental ballistie missile 
(ICBM). This phase offers the defensive system a longer time to track and intercept, and 
provides the possibility of having enough time to launch more than one interceptor. How¬ 
ever, this phase also supports the deployment of countermeasures against the defensive sys¬ 
tem. 

The Midcourse Defense Segment has two systems: the Ground-Based Midcourse 
Defense (GMD) [MDA02q] and the Sea-Based Midcourse Defense (SMD) [MDA02r] 
elements, successor systems to the NMD and NTW programs. The GMD interceptor has 
two major parts: the booster vehicle and the Exoatmospheric Kill Vehicle (EKV) 
[MDA02q]. The EKV destroys the target with kinetic energy, closing on the threat at a 
closing velocity of approximately 15,000 m.p.h. at altitudes more than 100 miles above the 
earth [MDA02q]. The SMD includes the existing Aegis Weapon Systems (AWS) with the 
AN/SPY-IB/D radar, and the Standard Missile 3 [MDA02r]. 

3. TERMINAL DEFENSE SEGMENT (TDS) 

The BMDS Terminal Defense Segment (TDS) [MDA02i, MDA02t, MDA03a] cov¬ 
ers the final phase of the threat’s flight that normally lasts approximately 30 sec. for an 
ICBM class of vehicle reentering the earth’s atmosphere at over 2,000 M.P.H. [MDA02j]. 
Countering the TDS threats are the PAC-3 missile [MDA02v], a high velocity HTK tech¬ 
nology against aircraft, cruise missiles, and short-range ballistic missiles, the Arrow 
Weapon System [MDA02t] and the MEADS [MDA02w]. The THAAD ground-based sys¬ 
tem complements the PAC-3 and is capable of engaging short-and medium-range ballistic 
missiles inside and outside the Earth’s atmosphere [MDA02u]. 

4, SENSOR SEGMENT 

The Sensor Segment [MDA02i] has two primary projects: Space Sensors and Inter¬ 
national Cooperation. The Space Sensors project supports the Block 2010 Space Based 
Infrared System (SBIRS) increment 3 and Space Tracking and Surveillance System (STSS) 
and will also support the BMDS Boost, Midcourse, and Terminal Segments with a constel¬ 
lation of infrared sensing satellites. The International Cooperative project, the Russian- 
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American Observation Satellite (RAMOS) [BMDOOa] is a cooperative research and devel¬ 
opment effort to observe the earth’s atmosphere and ballistic missile launches. The Targets 
and Countermeasures Program [MDA02s] provides threat representative targets and coun¬ 
termeasure challenges to the evolving BMD system and elements. 


5. BATTLE MANAGEMENT / COMMAND AND CONTROL (BM/C2) 


The BM/C2 [MDA02i] architecture will provide a highly complex and advanced C2 
element to effectively manage the system segments and execute the battle management 
function by achieving interoperability^"^^ with the system elements, external interfaces, and 
integrate with the national military command structure. BM/C2 improvements include al¬ 
gorithms and techniques to improve the control of the future missile defense system. 
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PAC-3 BMC2 

JCTN 

THAAD EMD 

UEWR 

ABL BMC4I 

JMMN 

GMD GBI1 W/EKV 

SEA-BASED XBR 

SBL BMC2 

DISN 

GMD GBI 11 W/EKV 

GBR/XBR 

JTAGS/ALERT 

IBIS 

AEGIS SM-2 BLOCK IV 

AEGIS SPY-IB 

MGS 


NAVY LEAP ALI 

ARROW RADAR 

THAAD BMC3 


SMD SM-3 

MEADS FIRE CONTROL 

MEADS BMC2 


ARROW 

MEADS 

MEADS SURVEILLANCE 

SATELLITES 

DSP 

ARROW BMC2 

TAOC 

CRC 


DIRECTED ENERGY 

SBIRS-HIGH 

PLATFORMS 


AIRBORNE LASER (COIL) 

SBIRS-LOW 

YAL-IA ABL (747-400) 


SPACE-BASED LASER 

RAMOS 

LASER 

THAAD LAUNCHER 
PAC-2 LAUNCHER 
PAC-3 LAUNCHER 


KINETIC ENERGY 

SEA-BASED KINETIC 
SPACE-BASED KINETIC 

BEACON ILLUMINATOR 
TRACKING ILLUMINATOR 
ACTIVE LASER RANGER 

AEGIS 



Table B-4. Major Ballistic Missile Defense Systems 
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Interoperability is the ability of systems, units, or forces to provide services to or to accept services from other systems, units, or 
forces and to use the services so exchanged to operate effectively together [MDA02b]. 
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D. BMD SYSTEM ACQUISITION STRATEGY 


The new Ageney mission is to develop, test, and prepare for deployment of a mis¬ 
sile defense system that will be able to engage all elasses and ranges of ballistie missile 
threats with eomplementary land-, sea-, air-, and spaee-based intereeptors, directed energy 
weapons, sensors, battle management, command and control, and communications systems. 
The BMDS capability identified in Figure B-5 will be characterized by: 

• A constantly changing system configuration, 

• Continuous upgrades to software, 

• A constantly changing threat, 

• Limitations imposed by the physical environment which must be understood, 

• Interoperability and integration challenges, and 

• The political will of the Nation. 

• The Agency’s has a three-phased evolutionary acquisition program [MDA02k]: 
development, transition, and procurement and operations [Rum02c]. A re¬ 
search, development, test, and evaluation program will support the fielding of 
missile defense capabilities to the field in two-year increments, with each suc¬ 
cessive block having a greater capability than the previous blocks. 



Figure B-5. BALLISTIC MISSILE DEFENSE SYSTEM (BMDS) - 2002 


350 






E. 


BMD SYSTEM TECHNOLOGY PLANNING 


Technology played a critical part in the previous missile initiatives. Future required 
advances in technology include many areas: sensors^"^^, weapons^^®, and BMC3. Critical 
sensor area research includes the Advanced Radar Technology program, which focuses on 
anticipating and negating countermeasures, improving discrimination functions, and aids 
the identification of the target in a potentially cluttered environment. The resolution of 
many complex countermeasure issues requires the development of more advanced dis¬ 
crimination algorithms and infrared sensor technologies. 

Future weapon technologies include research into Space-Based Lasers, Atmos¬ 
pheric and Exo-atmospheric Interceptor programs. These programs will incorporate ad¬ 
vanced HTK technologies, reduce costs, improve seeker capabilities and materials of the 
interceptor to discriminate and correctly identify the correct target in time to close and de¬ 
stroy the target. 


Sensors and detectors working individually or in networks may include passive sensors in the visible (optical), infrared, and ultravio¬ 
let ranges; active sensors including radars or ladars (laser detection and ranging); and may be airborne, ground-, sea-, or space-based. 
Sensors on interceptors and satellites are normally passive, while active sensors requiring high levels of energy are normally surface- 
based [BMDOOa], 
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Weapon systems may be airborne, ground-, sea-, or space-based, and must deliver enough energy into the target to destroy or negate 
it. These technologies currently include HTK interceptor systems and directed energy (laser) systems [BMDOOa] 
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APPENDIX C- COMMANDERS ANALYSIS AND PLANNING SIMULATION 

(CAPS) 


The CAPS simulation illustrated in Figure C-1 eonsists of the following eompo- 
nents configured in separate executable files: a graphical user interface (GUI), a simulation 
engine, and several utility programs, which convert data formats and support analysis. A 
user may enter the simulation environment or run the simulation through the GUI or via a 
shell script. The GUI program interfaces with the other CAPS components via their input 
and output files, detects program termination and displays the results. 

CAPS may simulate missile defense campaigns up to several months in length, with 
a scripted striking force, and a defending force that attempts to prevent damage to defended 
areas from the striking force ballistic missile threat (Figure C-2). Developed threat scenar¬ 
ios may consist of a series of raids against selected targets in specified regions of the world. 
A raid may consist of a mix of ballistic missile and air breathing threats (e.g., aircraft, 
cruise missiles), termed threat objects, available to the striking force. Threats may be vali¬ 
dated threats or user-created/modified. 


CAPS TODAY IS MDA’S 



Fast-Running, High-Level, Monte Carlo 
Constructive Simulation Supporting 
Development Of The Ballistic Missile 
Defense System - Employed Primarily 
As A High-Level Analysis Of 
Alternative And Planning Tool 



Depth Of 
Coverage 


CAPS Supports 
Planning Of Ballistic 
Missile Defense 
Alternatives And 
Trade Space 



CAPS Models Ballistic Missile 
Defense System Options For 

• Defense Footprints 

• Operating Area 



• Defended Area 

• Scenario Performance 

CAPS Supports First- 
Order Solution And Is 
HLA Compliant 


Fast-Running, Low To 
Medium Fidelity, Constructive 
Simulation With Diagnostic 
Levels, Graphic User 
Interface^ 

... Simulating Tactical Systems 


... Of The Ballistic Missile 
Defense System Elements (-) 


... In User Defined Simulated 
Environments 


... For Alternative Assessment 
And Planning 


... Simulating The Theoretical 
And Hypothetical Capability 


... Centrally Managed 
Verification, Validation, And 
Accreditation Program 


Figure C-I. CAPS Overview (From [GreOSa]) 
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System libraries for many current systems and regional topographic data are pro¬ 
vided, and a user may create additional systems tailored to the individual study require¬ 
ment. Topographic data libraries for selected regions are of medium-high resolution, while 
the entire world is provided at lower resolutions. 

Documentation for CAPS includes a User’s Manual supporting analysis procedures, 
the Methodology Manual [SpaOO] describes the methods and algorithms in the simulation 
engine, the Database Guide [SpaOl] provides the data structure and content of the CAPS 
database, and the CAPS Training Brief. The CAPS system also provides several sources of 
information, including a statistical summary on the effectiveness of the defensive and of¬ 
fensive forces, event logs from previous runs, and a status file that allows follow-on execu¬ 
tion of the campaign supported by CAPS utility programs for plotting and data extraction. 



Defense Operating Defended Scenario 


Figure C-2. CAPS Intended Uses (From [GreOSa]) 

The defending force deploys sensors and interceptors in space and the air, on land, 
and at sea, to detect and engage the threat objects before they can damage defended areas. 
Before each scenario, the defending force for each analysis option is created with the GUI, 
or modified from a previous scenario. The CAPS simulation engine reads all input files. 
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initializes the eampaign system eloek and all simulation eomponents, before entering the 
main simulation loop. 

Within the main simulation loop, CAPS is a diserete-event simulation and a hybrid 
event-driven, time-stepped eonstruetive simulation [SpaOO]. In the hybrid event-driven, 
time-stepped eonstruetive simulation mode, time-compression algorithms reduce run time 
in scenarios with several periods of intense activity, separated by long periods of relative 
inactivity prior to the next event. (Figure C-3). 

Using time compression, CAPS is able to run an 80-day theater-level campaign in 
approximately an hour, depending on the available machine resources. CAPS was de¬ 
scribed by [BMD99a] as a low-fidelity theater-level analysis tool, capable of modeling sys¬ 
tem performance and interceptor usage for land-, air-, and sea-based missile defense sys¬ 
tems employed against TBMs. 

In CAPS the primary simulation loop contains three segments (Status Update, Ac¬ 
quisition and Tracking, Engagements) and acts as the executive [SpaOO]. Within this loop 
the CAPS phenomenological models that simulate the campaign environment, threat ob¬ 
ject, sensors, weapons, and command authority functions simulate the events that occur 
within the current step [SpaOO]. 

The first segment in the CAPS Simulation Engine processes the status of all defend¬ 
ing units and all of the threat objects through a call to the CAPS Composite Threat Model. 
This process activates threat objects in the current time slot, updates a state vector with the 
threat object location and velocity, and deactivates threat objects which reach their target 
during the current time slot. The status of all defensive units are then updated by calls to: 

1) the CAPS Deployment Model for changes to sensor or interceptor locations; 2) the 
CAPS Airborne Interceptor Model, for the latest status of airborne interceptors; and 3) the 
CAPS Interceptor Model that provides the inventory adjustment status for interceptors 

The last task of this segment is a call to either the Phillips Eaboratory Airborne 
Simulation Tempest Revision (PLASTR) model or the ISAAC model, developed by the Air 
Eorce, that simulate the suite of sensors and the laser weapon of the ABE. This task is nec¬ 
essary due to the ABE’s autonomous search, acquisition, and tracking by the onboard sen¬ 
sors and engagement by the ABE laser weapon [SpaOO]. 
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• Graphical User Interface 

• Input And Output Databases Are Managed Through Tools Provided 
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Figure C-3. CAPS Architecture (From [Gre03a]) 

The second segment determines the acquisition and tracking of threat objects by de¬ 
fending unit sensors, according to the status of each sensor and each threat object. This 
segment is accomplished by calling the CAPS Radar Search and Acquisition Model to pro¬ 
vide the search, acquisition, detection, and tracking functions; followed by a call to the 
CAPS Command, Control, and Communications Model for communication link status; the 
CAPS Composite Threat Model to update state vectors for active threat objects; the CAPS 
Radar Signature Model to determine whether the defensive sensor systems actually detect 
threat objects in their field of view; and lastly, a call to the CAPS Threat Status Model to 
update the status of all active tracks and to initiate new tracks [SpaOO]. 

The final segment processes the engagement status of the threat objects based on 
the results of the previous two segments, calling the CAPS Battle Management Model to 
initiate new engagements, followed by a call to the CAPS Interceptor Engagement Model 
to update engagements in process or when it creates a new engagement. Both models call 
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the CAPS Command, Control, and Communications Model for the latest status about 
communication links. 

The CAPS model is mostly deterministic, with the exception of the stochastic radar 
detection model [BBG+99f]. CAPS currently runs under UNIX, Windows, and Motif on 
various host computers, and has been tested on Sun SparcStations, Silicon Graphics Indigo 
work stations, and personal computers running the Linux, Windows NT and Windows 
98/2000 [SpaOO], 
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APPENDIX D -MISSILE DEFENSE WARGAME AND ANALYSIS RESOURCE 

(MDWAR) 


The MDWAR synthetic battlespace or “gameboard” supports operators with a real¬ 
time constructive simulation of futuristic air and missile defense C2 systems in an inte¬ 
grated manner, allowing autonomous and interactive missile defense C2 simulations with 
variable resolution M&S for air, ground, space, and naval forces and their respective 
weapon systems, sensors and C2 [TRWOOb]. MDWAR, illustrated in Figure D-1 supports 
both HLA and DIS protocols allowing distributed execution and interoperability with real 
C4I systems [TRWOOb]. MDWAR provides models of all NMD core elements and spe¬ 
cific TMD models for all of the four JTAMD pillars of the active defense. 


MDWAR TODAY IS MDA’s... 


Virtual Operator-In-The-Loop For Concept Of 
Operations (CONORS), Doctrine, And Tactics, 
Techniques And Procedures (TTP) Development Using 
Operator-In-The-Loop (OITL) Perspective For 
Warfighters 



MDWAR Stimulates 
Discussion Of 
Missile Defense Issues 


MDWAR Supports Force-On- 
Force Architecture 
Assessment Analysis And 
Combatant Commander 
Assessments And Exercises 

HLA / DIS Compliant 




Real-Time, Medium 
Fidelity, Force-On-Force, 
Virtual, OITL Tool 


. Simulating Elements Of The 
Ballistic Missile Defense 
System 


. In A Realistic, Combat 
Environment 


Using Human-In-Control 
Experiments 


. Including Real-World 
Confusion From Fog Of War 


. Centrally Managed VV&A 
Program 


Figure D-1. MDWAR Intended Uses (From [GreOSa]) 

A goal of the MDWAR C2 war game process is to satisfy approved test objectives, 
identify the human usability of the missile defense simulation, and use this feedback to de¬ 
velop and refine the CONOPS, doctrine, TTPs, acquisition strategy and military utility of 
the evolving BMDS. The operator feedback supports the evolution of weapon designs, in- 
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teroperability, and operational procedures, balancing the effectiveness of the missile de¬ 
fense system with the capabilities of human operators to manage the system. A product of 
the MDWAR experiment process is the engineering data to support system designers, op¬ 
erational planners, and decision makers with future design decisions, CONOPS, TTPs and 
missile defense doctrine. 

MDWAR simulates a large number of objects ('-'1,000) in real-time, with sufficient 
detail and resolution to allow for discrete differences between alternative CONOPs options 
for operators, analysts, and decision-makers, and supports missile defense experiments in 
five different modes: 

• CONOPS Development - This is the primary mode. The remaining four modes are 
derivatives: 

• Combatant Commander (formerly CINC) assessment and field exercise, 

• Staff and operator familiarization, 

• Missile defense architecture assessment, 

• Test and Evaluation [TRWOOb]. 

MDWAR replaced the Advanced Real Time Gaming Universal Simulator (AR¬ 
GUS) model due to requirements for higher fidelity weapon and sensor models, more com¬ 
plex and realistic games with player positions similar to those in the field, and the need to 
reduce the turn-around time between games and analyze lessons learned. The MDWARS 
architecture (Figure D-2) is a distributed architecture, which allocates the supporting tasks 
to computational resources separate from the processors required to support the mission- 
space models. MDWARS is data driven, versus ‘hard-wiring’ or internetworking, employ¬ 
ing parameter files for missile, radar containing quantity, performance, location, and other 
data, upon initialization [TRWOOb]. 

In order to enhance the performance of the mission space models and the proces¬ 
sors, an optimistic parallel discrete simulation (PDES)^^^ framework, the Agency selected 
the Synchronous Parallel Environment for Emulation and Discrete Event Simulation 
(SPEEDES) as the simulation executive. To achieve performance objectives, SPEEDES 
executes in a host environment featuring processor clustering, network computing, and par- 


PDES simulations are categorized by how time is managed, how time management functions are distributed, if distributed, and how 
the distributed components interact [TRWOOb], 
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allel computing architectures employing Symmetric Multiprocessing (SMP)^^^ and Cache- 
Coherent Non-Uniform Access (CC-NUMA)^^^ systems [TRWOOb]. 
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Figure D-2. MDWAR’s Architecture (From [GreOSa]) 


The MDWAR architecture is composed of three sub-components called Top-Level 
Computer Software Components (TLCSC), which are executed as linked applications: 

• Missile and Air Defense Simulator (MADSim) TLCSC - This TLCSC is composed 
of the mission-space representations, the PDFS framework, and the interfacing 
gateways and simulator controls. 

• Viewers and Editors TLCSC - This component includes simulation control, analy¬ 
sis monitoring, and player displays supporting the human interface to MADSim 
during game execution. 

• MDWAR Resource Repository TLCSC - This component maintains all the data 
and documentation for all phases of the game, after action review (AAR) interfaces 
to the simulator’s object-oriented database, and analysis support tools [TRWOOb] 
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On a SMP, communication between processors is through shared memory and a very high -speed bus [TRWOOb], 

In a CC-NUMA architecture, the processors access shared memory through a cross-point switch, whereas the SMP shares both mem¬ 
ory and data buses [TRWOOb] 
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APPENDIX E - EXTENDED AIR DEFENSE SIMULATION (EADSIM) 


Since 1987 EADSIM, shown in Figure E-1, has evolved and today provides capabilities 
supporting fixed and rotary-wing aircraft, ballistic missiles, cruise missiles, sensors, command and 
control, communication jammers^^"^, and communication networks. EADSIM supports three stan¬ 
dard interface protocols— ALPS, DIS, and HLA— and may be run either in simulated or real time, 
and supports message passing, event passing, and control passing, via these standard protocols 
[BAC+95]. EADSIM is compatible with Unix operating systems on Silicon Graphics workstations. 
Sun workstations, and Windows NT [BBG+99e]. 


EADSIM TODAY IS MDA’s . . . 



System-Level Ballistic Missile 
Defense Constructive Simulation 
Modeling Multiple Functional And 
Physical Areas Of Interest 




EADSIM Models Flight 
Movement, 

Communications, Sensors, 
Jammers, And Weapons 






ur 




Supports Exercise 
Execution, And 
Pre- And Post- 
Exercise Analysis 

HLA, DIS, ALSP 
Compliant 



Medium Fidelity, Constructive / Virtual 
Simulation, Flexible, Composable, 
Tailorable, Efficient Simulation / 
Stimulation 



...Stimulating Tactical 
Hardware And Software 


...Of The Ballistic Missile 
Defense System Elements (-) 

...In Realistic, Simnlated 
Environments 

...Eor Interoperability 
Assessment And Integration 

...Emulating The Joint Data 
Network And Comms 

...Centrally Managed Verification, 
Validation, And Accreditation 
Program 


Figure E-I. EADSIM’s Intended Use (From [Gre03a]) 

EADSIM architecture in Figure E-2 includes the following general models: air defense, of¬ 
fensive air operations, attack operations, multi-stage ballistic missiles, air breathers, sensors, jam¬ 
mers, satellites, early warning, generic noncombatants, communications, electronic warfare, and 
specific areas of interest [BAC+95]. The simulations flexibility is a key reason for the major role 
EADSIM plays in the BMDS cooperative international M&S program. EADSIM performs three 
basic functions: simulation set-up, scenario execution by a set of run-time models running in a 
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Jammers are radio transmitters accompanying attacking RVs and tuned to broadcast as the defensive radar to add ‘noise’ to signals 
reflected from the RV and received by the defensive radar [MDA02b]. 
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multi-process mode, and post-processing and analysis. The EADSIM simulation also features an 
integrated graphic user interface for set-up and post-processing support [BAC+95]. EADSIM ex¬ 
plicitly models each platform. Modeled platforms represent a system type (e.g., PAC-3) and may 
have any number of elements such as sensors, weapons, defined by the user [JAS97c]. 


■ Integrated Graphical User Interface 

■ Input And Output Databases Are Managed Through Tools Provided 

■ Interface To Other Models Via DIS / HLA / ALSP 
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Figure E-2. EADSIM’s Architecture (Erom [Gre03a]) 

EADSIM is a Monte Carlo model with a capability able to support some determi¬ 
nistic analysis. A set of run-time models represent selected aspects of the scenario, support 
scenario execution, and perform data exchanges during execution [BAC+95]. There are 
four EADSIM run-time processes: C3I, flight processing, detection, and propagation 
[TOM97]. Major functions in EADSIM include the following: battle management, com¬ 
munications, movement, propagation, IR and radar signatures, radar sensors, and natural 
environments. All models in EADSIM rely on data files for storage and retrieval of sce¬ 
nario information, and reflect the level of model abstraction maintained in the data files 
[Tom97]. 
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APPENDIX F - EXTENDED AIR DEFENSE TESTBED (EADTB) 

EADTB provides a capability to invoke pre-compiled algorithms with an inter¬ 
preted code, which provides the user with a significant capability to control model func¬ 
tionality and define the weapon or system under study [BBB+96]. EADTB, a two-sided 
model (Red and Blue forces), illustrated in Eigure E-1, models many types of air defense 
weapon systems, with algorithms supporting the varying degree of representation detail 
required; and features the ability to model perception versus truth data, that allows the in¬ 
corporation of limited, flawed, or misinterpreted information to influence the decision 
process [BBB+96]. Missile defense functional areas modeled by EADB include; BM/C2, 
movement, missile launching, lethality, vulnerability, signature generation, communica¬ 
tions, and the natural environment [Ray02a]. With its variable levels of detail, it is able to 
support studies of systems, components, tactical doctrine development, and mission plan¬ 
ning [BMD99a]. 


EADTB TODAY IS MDA’s . . . 


Perception-Based 
Constructive Simulation 



Perceived Data That Contains Errors 
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Performance In Real-World 
Conditions 
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Figure F-1. EADTB’s Intended Uses From [GreOSa]) 
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SSRs, called rule sets, are a major feature of EADTB’s eapability and flexibility, al¬ 
lowing EADTB to funetion at various levels of detail and aggregation [BBB+96, MROl]. 
The EADTB Common Model Set (CMS) maintains SSR algorithms, and provides baek- 
ground proeesses and routines ealled by SSRs [BBB+96]. User-developed SSRs, built 
from EADTB components (e.g. Thinker, Platform, Communieations, and Sensor) represent 
specifie weapon systems supported by the modeled natural environment in whieh these 
eomponents operate. 

Eaeh SSR has at least one or more Thinker eomponents [Wai94, WS94], whieh 
provides the intelligenee for the SSR, the eonstruetive surrogate for the human eommand 
and eontrol. Thinker rule sets aet on pereeption^^^ data as opposed to truth^^^ data. Thinker 
algorithms are available to rule sets through a set of speeifie symbols. The Thinker re- 
eeives data from onboard sensors and by way of the eommunications network. However 
EADSIM proeesses “truth” data by sending eommunieation, sensors, and other elements 
information transmitted to the Thinker as “pereeived” data. Integral to the Thinker are the 
user-defined Rule sets [ZCE+01]. 

The rule set logie eombines with the representation of the eommand and eontrol 
system and available eommunieation networks to simulate the C2 proeess. The Rule set 
invokes a series of algorithms to eomplete functions including threat assessment, missile 
guidanee, eorrelation, fusion, and weapon assignment. EADTB supports a wide range of 
potential SSRs supported by user-defined rule set logie, which may represent both manual 
and automatie proeesses. The EADTB interpreted rule set in EADTB eonsisting of a set of 
symbols and syntax rules provide all C2 deeision-making logie, and supports simulation 
seenario ehanges without reeompiling or linking. The flexibility inherent in EADTB and 
the rule sets also ereates the greatest ehallenges, since it requires so mueh input from the 
users of the tool. 

A SSR also has one Platform eomponent. The Platform eomponent eontains algo¬ 
rithms to allow it to model movement and other physieal characteristies of different Plat- 
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‘Perception’ data constitutes the results of perception transformations (observations, derivation, or estimates) by given entities of the 
states of other entities within the scope of the simulations / exercise as manifest in local perceived-information data stmctures (possibly 
corrupted, inaccurate, imprecise, or imperfectly known) [Wai94] 

‘Truth’ data is generated as part of the original, intrinsic unequivocal representation within the scope of the simulations / exercise and 
which constitutes the truth regarding the states of the entities as known to themselves or the executive [Wai94]. 
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form types, different kill meehanisms, Platform signatures, and Platform vulnerabilities. 
The Platform component has five constituent capabilities or physical characteristics that 
correlate to the real world: 1) move / motion capabilities, 2) the ability to cause damage, or 
3) be damaged, 4) the ability to carry and/or launch an object, and 5) the ability to generate 
a signature for reflection or emission [Ray02a]. 

A Communication component allows the transfer of data between Thinkers, but 
does not access the contents of the message. Determining who may communicate is dy¬ 
namic since the operational state of communication devices may change. However, 
EADTB delivers only error-free messages from the receiving communication unit to the 
receiving Thinker. Two communication models support message transfers, a low-detail 
model called Direct Comms for statistically modeling message delay, and the Standard 
Comms model which models the links, queues, networks, and propagation factors 
[Ray02a]. The two methods for dissemination messages are point-to-point and broadcast. 

Sensor components model the detection of objects by measuring energy levels re¬ 
flected or emitted by the object. Target detection is a function of the sensor performance, 
target-signature characteristics, and environmental effects. The user defines the sensor per¬ 
formance by selecting the sensor power and gain. Sensor models may be active sensors 
such as radar^^^ or passive sensors including EO/IR passive^^^ and RE passive sensors^*’*^ 
[Ray02a]. A sensor, controlled by rule set commands, and tasked by a Thinker, may 
transmit and/or receive over a common set of user-defined extended frequencies. 

The level of sensor detail ranges from functional sensors that only process range, 
target type, and search volume; to fine-grained sensor models that can model gain, 
polarization, Doppler spreads, ambiguous ranges, and jamming (main lobe or side lobe) 
[Ray02a]. The Sensor component is comprised of a set of models used to simulate the fol¬ 
lowing sensors: radar, RE passive, RE noise jammer, EO/IR passive, and functional sensor; 
which are further decomposed into four subcomponents: scanner, transmitter, receiver, and 
data processor [Ray02a]. 


The signature may be in the radio frequency (RF), electro optical (EO) or infrared (IR) wavelengths. 
Radars transmit RF energy and measure its reflection from the body of the target [Ray02a]. 

™ Sensors that measure the energy emitted by the body of the target [Ray02a]. 

Sensors that measure the energy emitted from an active sensor transmitter carried on the target [Ray02a]. 

367 



The Environment algorithms model the portion of the environment that affects ob¬ 
jects by three feature groups: 1) terrain and sea surface, 2) atmosphere, and 3) celestial bod¬ 
ies. The simulation supports DTED, DEAD and user-derived data sets [Ray02a]. The en¬ 
vironment influences sensor and communication representations, sensor detection, and the 
performance of object movement representations. 



Specific System Representation (SSR) ^ 


EADTB Entities Are Built From Components, Allowing The User 
The Flexibility To Create Representations Of Systems At The Detail 
Of The Component Pieces 
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Rulesets Are Isolated From One Another So 
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Paradigm 


EADTB Is DIS And HLA Compliant - Additionally Provides The Ruleset 
Interfaces To External Processes For BM/C2 SWIL 


Figure F-2. EADTB’s Architecture (From [Gre03a]) 

EADTB analysts build scenarios incrementally, developing elements such as air¬ 
craft, communication devices, and sensors. When deployed on a digitized terrain map, the 
systems become platforms, and then combined into lay downs, and lastly, merged into sce¬ 
narios. EADTB architecture shown in Figure E-2, runs on a quad-processor Covex 3840, 
supporting Silicon Graphics workstations and successfully ported to Silicon Graphics and 
Origin 2000 workstations. EADTB is currently DIS and HLA compliant [SAP98, Wai99, 
STB+00, WSOO]. 
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APPENDIX G - MISSILE DEFENSE SYSTEM EXERCISER (MDSE) 

Starting with a proof of principle in 1993, and evolving with an ineremental build 
methodology, MDSE, shown in Figure G-1, exereises real tactieal hardware and software 
in dynamic, interactive interoperability tests allowing the hardware to be operated earlier in 
the development cycle and over a wider range of conditions than may ever be possible in 
live tests. A simulated secure voice radio environment between tactical operators at each 
site support MDSE HWIE tests, while test operators have secure voice communications 
coordination with recording and playback capabilities. MDSE uses the DIS protoeol 
[MeQ97]. 


MDSE TODAY IS MDA’s . . . 



Ballistic Missile Defense 
System-Level Distributed 
Real-Time Hardware-In-The- 
Loop (HWIL) Tool 



MDSE Supports Event HWIL Planning, 
Control, And Assessment 




MDSE Monitors HWIL 
Troth, Situational 
Awareness, And 
Activity Data 




MDSE Supports 
HWIL Post-Test 
Analysis 


t /St 


DIS Compliant 


Real-Time, Virtual 
Simulations, Centrally Controlled, 
Geographically Distributed, 
^PS Synchronized HWIL To(^ 


. Stimulating Tactical 
Hardware And Software 


. Of The Ballistic Missile 
Defense System Elements (-) 

. In Realistic, Simulated 
Environments 


. For Interoperability 
Assessment And Integration 


. Emulating The Joint Data 
Network And Comms 

. Centrally Managed (SES) 
Verification, Validation, 
Accreditation Program 


Figure G-1. MDSE’s Intended Uses (From [Gre03a]) 

As a virtual model, MDSE’s objeetive is to stimulate the actual missile defense 
hardware and software to the maximum extent possible and employ embedded eonstruetive 
simulations only when neeessary due to eost, availability, or physios. Areas where oon- 
struotive simulations in MDSE’s arohiteoture (Figure G-2) are employed inolude; the infra¬ 
red and radar sensor front ends; transmitter, reoeiver and signal prooessing; interoeptor fly 
out; and portions of the BMC3 [MoQ97]. MDSE also emulates the Joint Data Network 
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(JDN) [YSOO] with a gateway in order to pass Taetieal Digital Information Link-J (TADIL- 
J) [DoD97, BSB+02], Taetieal Information Broadcast System (TIBS), Tactical and Related 
Applications (TRAP), and Tactical Data Distribution (TDDS) tactical messages to exercise 
participants [Too94]. 

MDSE then stimulates the sensor data processing systems of the tactical software 
and hardware to develop a flow of information through the tactical system interfaces. The 
stimulation supports an assessment of FoS interoperability issues [BB98a, BB98b], includ¬ 
ing reporting responsibility (R2), track management, cueing effectiveness, engagement co¬ 
ordination, and tactical battle management and C2 implementation in the participating sys¬ 
tems [Too94]. The stimuli for the track processing, based on realistic representations of 
pre-scripted theater-level attack scenarios, pass over a geographically distributed network 
to a remote-environment interface connected to the tactical system [WasOl]. 

MDSE is composed of three segments interconnected by the Test Communications 
Network (TCN), the MDSE Control Segment (TCS), the Tactical System Driver Segments 
(TSDs), and the Tactical Communications Environment Segment (TCES). The TCN con¬ 
sists of a combination of MDSE component and segment local area networks and wide area 
networks interconnected by T1 encrypted lines in a star topology between the TEC at the 
hub to all the remote environments (RE) at the test sites [McQ97]. The TCN allows MDSE 
to monitor and affect the health and welfare status of the MDSE network. 
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■ Gateway Interfaces / T-1 Circuit 

■ Intra-Segment interfaces 

- Test Control Interfaces / T-1 Circuit 
Blue Text Constructive Model 


Build 3,4To4,0/4,l 
_ Upgrade 



Figure G-2. MDSE’s Architecture (From [Gre03a]) 


The Tactical Communications Environment Segment (TCES) emulates the Joint 
Data network (JDN) and enables the tactical systems to exchange tactical messages consist¬ 
ing of emulated TADIL-J communication link messages in a simulated communication en¬ 
vironment, that result from the injection of truth data into the various systems [TEMOO]. 
The TCES consists of the Link-16 Gateway System for JDN emulation and emulation of 
the TIBS and TDDS messages. The Link-16 Gateways also provide the TEC with message 
display, monitoring, and recording capabilities with a Central Node Gateway, at the HWIL 
test control site, and a Remote Node Gateway at each one of the participating test sites. 

The MDSE TCS segment simulates the environment of the operational architecture 
and provides overall control of the HWIL test. The TCS consists of a Test Exerciser Con¬ 
trol (TEC) at the HWIL test control location, and an RE at each of the participating test 
sites. A Theater Environment (TE) node, a real-time simulation of the physical environ¬ 
ment within which the tactical systems will operate, is located within the TEC and at each 
RE. Physical phenomena include threat object propagation and debris, system location. 
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propagation of natural objects such as weather, terrain, natural and hostile environments 
and auxiliary interfaces such as the global positioning system (GPS). 

A major function of the TE node is to provide tactical systems only that information 
which the systems would naturally perceive through their sensors or which affect their op¬ 
erational performance. By segregating simulation truth data from tactical system percep¬ 
tion, MDSE ensures that that a tactical systems performance is not biased by information 
that would not be available to the tactical system under the simulated scenario, since the 
tactical system must react to the TE node stimulus as it would to a real-world situation 
[McQ97]. 

The MDSE TSD segments accept truth data from the TCS Theater Environment 
(TE), determine whether the sensors can see the object(s) as scripted in the scenario, and if 
appropriate, convert the truth data into signals compatible with the systems sensor. The 
TSD segment consists of the tactical driver, which interfaces between the REs and the tac¬ 
tical system or simulator. The MDA program office responsible for the tactical system de¬ 
velopment provides the system driver through which MDSE interfaces, stimulates, and 
controls the test execution for the tactical system. 

The system drivers provide the physical interface to the system under test, receiving 
data from the MDSE TCS segment, and convert the data to a format usable by the tactical 
system. System drivers may also provide simulations of tactical components of the weapon 
system or sensor, required by the scenario, but may be missing or unavailable for a HWIL 
test [McQ97]. The hardware for the development and operational environments includes 
Sun Ultra’s, Sun Servers, Silicon Graphics Indigos, and X-terminals. 
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APPENDIX H - ELEMENT LEVEL M&S 


Table H-1 is a listing of the major Element-level M&S systems to provide a eontext 
of the constructive and HWIL Elements M&S systems within the MDA M&S Hierarchy. 


BMDS ELEMENT LEVEL CONSTRUCTIVE M&S 

PATRIOT 

THAAD 

GMS 

SMS 

ABL 

KEB 

PAC2SIM 

ETESIM 

LIDS 

JHU-6 DOF 

ISAAC 

TRITON E2E 

PAC3SIM 

TISES 

SENTRY 

KWEVAL 

LAMDA 

KWEVAL 

DOSE 

MEDUSA 

SSTB 

HYDROCODES 

SIMPROCESS 


MFSIM 

THHADSIM 

TESS 

MEDUSA 

BMAP 


FMS-D 

STAX 

TEX 

SSTRESS 

ICONOS 



TEAM-2 

RELEX 

ACCSIS 

ABLPROP 



SEEKER 

SIMTAS 

FIRMTRACK 

HITS 



ASTAB 

XBR DIGSIM 

ARTEMIS-T 





UEWR DIGSIM 

MARS 





UTB 






NMDSIM 






WPNSIM 




BMDS ELEMENT LEVEL HARDWARE-IN-THE-LOOP (HWIL) 

FMS 

SIL 

ISTC_I 

GSEL 

TACCSF 


GTSF 

TEC 

ISTC_2 

RAY HIL/CIL 



MSS-3 

TRTB 

PCIL 

CSEDS 



FMMFC 

TTC 


ETEDDS 




TMTD 





[PATOll 

[THA02] 

[TCD+OOI 

1AEG02] 

[ABLOI] 

1KEB02I 


Table H-1. MAJOR HMDS EEEMENT EEVEE M&S 
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APPENDIX I - THREAT, ENVIRONMENT, SIGNATURE, and LETHALITY 

(TSEL) MODELS 

Appendix I provides brief deseriptions of the BMDS System-Level Threat, Signa¬ 
ture, Environment, and Lethality M&S. These models support the development and testing 
of the Element-Level M&S listed in Appendix H. and the BMDS System-Level Core 
M&S. 

A, BMDS SYSTEM-LEVEL THREAT M&S 

1. TM93 

TM93 is a modeling and simulation architeeture used by the SPC to provide intelli- 
genee eredible representations of threat missile, aircraft, and cruise missile systems. The 
TM93 Threat Model is the official set of tools used within the Special Program Center 
(SPC) to generate threat files of ballistic threats and the responsibility for assuring that the 
threat scenarios used in simulations, studies, and analyses adequately represent the best as¬ 
sessment of the intelligence community. 

The SPC has the responsibility for the integration and execution of the various tools 
used to generate standard threat tapes. The TM93 Threat Model is the set of integrated 
tools used to produce standard threat tapes and includes tools to allocate strategic and thea¬ 
ter ballistic threats and air-breathing threats. The allocation tools feed tools, which gener¬ 
ate high fidelity trajectories of all threat objects including boosters, tanks, PBV's, RV's, de¬ 
bris, etc. Post-processing tools perform quality control checks and to feed the final docu¬ 
mentation process. All threats include a complete set of hardcopy documentation of the 
scenario. TM93 provides representations of single events through mass strategic attacks. 
TM93 models threat trajectories using an ECI coordinate frame, J4 oblate earth model, and 
standard atmosphere. 

The TM93 documentation includes a User’s Manual. TM93 uses a UNIX based 
operating system. However, certain applications only operate on IRIX/Silicon Graphics 
platforms. A conversion tool exists allowing data conversion to DIS PDUs. TM93 will 
meet HLA standards in the future. 
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2, STRATEGIC AND THEATER ATTACK MODELING PROCESS 

(STAMP) 

STAMP is a workstation-based threat generator and engineering analysis tool that 
models ballistic missile flights from launch to impact. STAMP is a missile flight generator 
and engineering analysis tool that models missile flight from launch to impact and presents 
extensive flight characteristics and trajectory descriptions using a wide array of graphical 
and tabular outputs. It features: an easy to use operator interface using Windows. 

STAMP is missile-database driven, developed and approved by intelligence agen¬ 
cies and contains the parameters and values needed to model strategic, theater, and space- 
launch missiles consistent with intelligence estimates. These databases are in a standard 
format, read directly into STAMP. The main output from STAMP consists of state vectors 
in a specified format. 

STAMP has three primary applications: (1) missile analysis; (2) scenario genera¬ 
tion; and space-launch vehicle modeling. The modeling and analysis tool assists with mis¬ 
sile definition development, checkout, and depiction of missile characteristics and flight 
trajectories to include penaids and sub munitions. The scenario modeling and analysis tool 
generates simple or massive attack scenarios and provides graphical and statistical charac¬ 
teristics of the scenarios. The space-launch vehicle-modeling tool allows users to specify 
the desired satellite orbit. STAMP will determine the missile flight path to achieve this or¬ 
bit. 

STAMP has a wide range of missile modeling capabilities that satisfy the needs of 
most users for accurate and timely missile performance information in functional formats. 
The plotting and data outputs from STAMP have evolved over several years of threat 
analysis support and have been continually refined to meet user needs. The primary appli¬ 
cations are: (1) Missile analysis, (2) Scenario generation, and (3) Space-launch vehicle 
modeling. 

Documentation includes the STAMP User’s Manual, STAMP Design Document, 
Missile Definition Data Base (MDDB) Files, Worldwide Ballistic Missile Baseline Per¬ 
formance Characteristics, ACE User’s Manual, and NTF Interface Requirements Specifica- 
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tion. STAMP source code includes Ada and C, and it runs on UNIX platforms (e.g., Sun 
and Silieon Graphics), with two support graphics packages (XMGR and AXIS). 

The following are inputs to STAMP; Missile Definition Data Bases (MDDB); faeility files; 
scenario specifieations; modeling parameters; earth model; Digital Integrated Combat 
Evaluator (DICE) system files; Modular Vehicle Simulation (MVS) aeronautical data. In 
addition to the graphie and tabular outputs, STAMP ean also generate every object threat 
files and MEM threat input files. 

STAMP links to the following models: TISES (THAAD System Simulation); PTOS 
(PATRIOT Tactical Operations Simulation); BESIM (Brilliant Eyes Simulation); PERM 
(Penaid Effectiveness and Requirements Model); MEM (Mission Effectiveness Model); 6 
DOE free flight models (ATAP & MVS); Commereial Satellite Analysis eodes (STK and 
OMNI); chemical and biological codes. The Space Warfare Center also uses STAMP to 
model all ballistic and space-launched missiles for input to their sensor. 
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B, HMDS SYSTEM-LEVEL SIGNATURE AND ENVIRONMENT M&S 

1. SYNTHETIC SCENE GENERATION MODEL (SSGM) 

The SSGM generates eomposite Infrared (IR) scenes, X-band radar cross sections 
and IR LIDAR cross sections and range/Doppler returns using high fidelity, validated stan¬ 
dard component phenomenology models. The SSGM consists of computer software and 
input databases that provide the capability to generate two-dimensional time-sequenced 
scenes as well as other supporting data for use in the design, development, and testing of 
surveillance and weapons systems. 

The SSGM computes viewer-perspective apparent-radiance pixel maps and point- 
source intensity data from the output of state-of-the-science phenomenology codes and au¬ 
thenticated input databases. This SSGM performs via an interactive software system that 
assists users in creating the scene-sequence scenario and content, selects or generates the 
required databases, and renders the databases to generate the desired output. In addition, 
provided tools allows users to display and analyze the output images. The standard wave¬ 
length regime extends from the UV to the far infrared, plus the extension to include hard- 
body-modeling capability at radar frequencies. The space-time samplings are those of the 
sensor systems and are not otherwise constrained. 

The SSGM developers restructured the software in Release 6.0 to make it easier to 
add and remove capabilities from the overall system. The SSGM uses the concept of a 
“component” to describe the different phenomenology model capabilities. The SSGM has 
five types of components: general, foreground, background, laser foreground, and non¬ 
imaging. General components include the Observer, Environment and Trajectory; which 
provide information to the models as required. The various phenomenology models in¬ 
clude foreground, background, and laser foreground or non-imaging components. 

SSGM radiance output scenes consist of quiescent and enhanced natural and per¬ 
turbed backgrounds with embedded foreground elements such as targets and target related 
events. Background components include terrain, horizon, earth limb, aurora, space, and 
nuclear events. Foregrounds include boosting missile plumes and hardbodies, and post¬ 
boost hardbody objects including RVs and decoys. Clouds maybe modeled as either back- 


378 



ground or foreground components. The hardbody LIDAR model is the laser foreground 
component. All radar components are non-imaging. 

SSGM contains numerous first-principles physics models. As such, it contains a 
number of assumptions and approximations. The principal limitation is the spectral cover¬ 
age and spatial resolution. Most SSGM imaging components cover the 0.2-25p-m wave¬ 
length range. Plumes and nuclear phenomena are restricted to a 2-25pm range. Most im¬ 
aging components support a spatial resolution of >10-9 radians. Celestial backgrounds are 
limited to >4 x 10-5 radian resolution. Terrain and cloud databases have inherent resolu¬ 
tions that vary from 30 - 18500 m and 30 - 1000 m, respectively. These effectively bound 
sensor footprint resolution. Primary applications include: 

• Sensor Development 

• Sensitivity & Trade-Off Analyses 

• Threat Signature Development 

• Nominal Threat Signature, Launch through impact 

• Realistic range and distribution of signatures 

• Algorithm Development 

• Fusion 

• Detection, discrimination and tracking 

• Data Analysis 

• Test and Evaluation 

• Test planning and excursion evaluation 

• Hardware-in-the-loop simulations and emulators 

• High-level simulations 

• Support for Wargames: signatures and backgrounds 

• System Architecture Studies 

The SSGM main architecture shown in Table I-l includes OOA/OOD techniques 
and C++. Phenomenology component codes include a mix of FORTRAN, C and C++. 
Interface between the component codes and SSGM involves C++ “wrappers”. The GUI 
written in C++, uses X-Windows and X-Motif call. Component model source code is gen¬ 
erally not available for modifications, as any modification might negate the validation 
status of the model. The Hardware includes Silicon Graphics workstations or servers run¬ 
ning IRIX. 
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Scenarios can take seconds to days to execute to run to completion. This is princi¬ 
pally a function of the number and complexity of the targets, types of background (e.g., 
clouds, earthlimb), phenomenology models selected and number of scenes to be generated. 

Current Inputs include Scenario Description File (all other necessary databases are 
internal). This includes sensor, target and engagement description, but does not include 
specific target trajectories. Current Outputs include 

• At-aperture radiance scenes 

• Target and plume radiant intensities 

• Target and sensor metrics 

• Target heating and ablation history 

• Detailed information on atmosphere, clouds, and terrain 


Atmosphere 

Nuclear 

SHARC 


IRSim 

MODTRAN 


HiSEMM 

SAMM 


PEM 

SAG 


Celestial 

APART 



MOSART 


IRAS Star Catalog 

FASTLIMB 


CBSKY4 

CIRRIS-1A 


CBZODY 



CBAMP 

Clouds 


TROs 

CLDSIM 


osc 

Aurora 

OPTISIG 

AURORA 


Xpatch 



DELTAS 

Terrain 


Missile Plume 

GENESSIS 





SIRRM 

Vents/Spills 

Debris 

CHARM 

Fuel Vent 

KIDD-OPTISIG 

CAMERA 

Vent-Trail 

DEBRA-KIDD 

SPF-2 


Table I-l. SSGM Constituent Model Matrix 
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2, THE BATTLESPACE ENVIRONMENT SIGNATURE TOOLKIT 

(BEST) 

The Battlespace Environment Signature Toolkit (BEST) program will be developed 
to replace the legacy Environment and Signature models shown in Table I-l and Table 1-2, 
establishing the foundation for the next-generation missile defense Scene Generation capa¬ 
bility for future advanced sensor development based on requirements identified by 
[CKB+94, Hec99, Hie99, SB99, BerOOa, BMC+00, BSB+00, BurOO, BVW+00, CWM+00, 
DKC+00, DNC+00, DSHOO, DWE+00, EAK+00, KMM+00, MSB+00, PBOO, SST+OO], 
The MDA awarded a contract in June 2002 for the development of the Battlespace Envi¬ 
ronment and Signatures Toolkit (BEST) development program supporting the design, de¬ 
velopment, testing and evaluation of active and passive sensors, signature processing, and 
algorithms with empirical and physics-based signature phenomenology and battle space 
environment codes for all threes phases of missile flight. 

Meeting the requirement to counter the full range of future potential threats, the 
MDA developed an Adversarial Capability methodology which will be introduced into the 
MDA System Core Models, augmenting the existing DIA validated Design-to-Threat, pro¬ 
viding BMDS system engineers with new capabilities for trade space analysis. 


BMDS Legacy Environment and Signature Codes 

Model/Code 

Function 

Executing Agent 

Target Related Hardbody Objects 

osc 

Resolved Passive EO/IR Signatures 

SMDC 

OPTISIG 

Unresolved Passive EO/IR Signatures 

SMDC 

XPATCH 

Radar Signatures (Physical Optics) 

AFRL/SN 

FISC 

Radar Signatures (Method of Moments) 

AFRL/SN 

RTS 

Radar Signatures 

NRL 

DELTAS 

Laser Retroreflectance 

SMDC 

Missile Plumes & Trails 

SFM, VIPER, TDK 

Flowfield, Nozzle & Combustion 

AFRL/PR 

SPF, GIFS 

Complex Plume Flowfield 

AFRL/PR 

PERCORP.SPP 

Liquid & Solid Motor Performance 

AFRL/PR 

SIRRM 

Plume Radiance (< 50 km) 

AFRL/PR 

CHARM 

Plume Flowfield & Radiance (> 50 km) 

AFRL/PR 

SPURC 

Plume UV/VIS Radiance (> 50 km) 

AFRL/PR 

SOCRATES 

Monte Carlo Flowfield & Gas Dynamics 

AFRL/PR 

HALT, PARCS 

Laser backscatter & Plume RCS 

AFRL/PR 
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Battlespaee Environment 

MODTRAN 

Atmospheric Transmission & Radiance (<50 km) 

AFRL/VS 

SHARC 

Quiescent Atmosphere, Auroral & Terminator Radi¬ 
ance and Clutter (>50 km) 

AFRL/VS 

SAMM 

Atmospheric Transmission & Radiance (all altitude) 

AFRLWS 

SAG, MSIS, BEAM, SIG 

Atmospheric State and Clutter 

AFRL/VS, NRL & MSIC 

CBSD 

Planets, Moons, Asteroids, Zodiacal Emission, Stars 

AFRL/VS 

GENESSIS 

Terrain & Ocean Surface Radiance 

AFRL/VS 

CLDSIM 

Cloud Radiance, Transmission, & Glints 

AFRL/VS 

IRSIM, HiSEMM, NucSim, 

NORSE 

Nuclear Perturbed Atmosphere 

DTRA 

Trajectory 

STAMP 

Strategic Missile Trajectories 

DIA/NAIC 

DICE 

Theater Missile Trajectories 

DIA/MSIC 

Integrated Models 

SSGM 

Targets, Trajectories, and Environments 

NRL 

PLEXUS 

Expert System for Atmospheric & Celestial Phe¬ 
nomenology 

AFRL/VS 

CHAMP 

Missile Plume & Hardbody 

AFRL/MN 


Table 1-2. BMDS Signature and Environment Funetions 


Beeause of the distributed nature of BMD weapon systems and the eritieal impor- 
tanee of eaeh element to aehieve overall system effeetiveness, interoperability and eollee- 
tive system performanee assessments are an important area of evaluation. All BMD sensor 
and weapon systems require signature information upon whieh to base deeisions. Existing 
MDA signature eodes have served the engineering analysis eommunity over the years, but 
they are diffieult and expensive to modify, diffieult for the user to learn and operate eor- 
reetly, and most importantly eannot effeetively support the future M&S needs of MDA. 
Present and antieipated requirements will demand greater interoperability, eonsisteney, and 
faster exeeution speeds..BEST will enhanee signature tools and eomplete the phenomenol¬ 
ogy set effeetively providing a robust signature toolkit for all MDA signature needs. It 
will: 

• Provide a more eomplete representation of battlespaee signatures. 

• Ensure eonsisteney in signature results thereby assisting verifieation and validation 
and aeereditation. 

• Meet the future ehallenge of simulation-driven design, evaluation, and aequisition. 

• Provide users the flexibility they require 

• Support interoperability (within the signature eodes and with external simulations). 
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BEST will replace the patchwork assemblage of signature codes with a consistent, 
modular code that is accessible through one interface. BEST will support the entire range 
of missile defense signature needs. The modular nature of BEST will support plug-and- 
play of functionally equivalent components. This will permit contractors to insert proprie¬ 
tary components and to even use the components delivered with BEST in other applica¬ 
tions. The following illustration depicts the component-based structure of BEST. 

A secondary goal of BEST is to provide the government with signature tools as part 
of a suite of government-approved tools to independently evaluate and assess contractor 
performance. 
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C. BMDS SYSTEM-LEVEL LETHALITY M&S 

The BMD System lethality model program consists of PEELS and PEGEM as a 
suite of tools continually evolving to identify any residual threat after a successful inter¬ 
cept. The BMD System lethality programs directly support the development of the MDA 
element level M&S and system level testing. 

1. POST ENGAGEMENT GROUND EFFECTS MODEL (PEGEM) 

The Post-Engagement Ground Effects Model (PEGEM) is a simulation tool to study 
ground effects caused by chemical, biological, and conventional weapons/agents distrib¬ 
uted in both unitary (bulk) and submunition (canister or bomblet) form. The model pro¬ 
vides chemical/biological agent ground contamination and conventional weapons blast 
zones. It also provides data for unit effectiveness and casualty estimation. 

PEGEM is a suite of models to produce a simulation incorporating various phases 
of ground effects assessment, such as Munition Propagation and Atmospheric Transport. 

In a typical case, the analyst specifies a chemical or biological weapon event scenario in¬ 
cluding all threat details and the locations and times of the various events on the user- 
defined grid. Intercept lethality information flows from the output of an endgame lethality 
model. The lethality model provides PEGEM a prediction of the fraction of payload sur¬ 
viving following an intercept event. Eor canister submunition payloads, the locations of 
surviving submunitions within the target payload are given. PEGEM uses this information 
to propagate the potential residual threat(s) to the ground. 

PEGEM documentation includes a User’s Manual, Technical Manual, and other 
supporting documentation. Written in ANSI-C, PEGEM runs on Windows 95, Windows 
NT and UNIX 
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2, PARAMETRIC ENDO- EXO-ATMOSPHERIC LETHALITY 

SIMULATION (PEELS) 

PEELS is a high fidelity terminal endgame lethality simulation, whieh models hit- 
to-kill and fragment kill of ballistie missile from kinetie energy weapon attaek. It operates 
as a non-real-time simulation, intended to support weapon system designers as well as sys¬ 
tem level analysts. The models primary purpose is to analyze lethal effectiveness of non¬ 
nuclear kill interceptors in an anti-tactical ballistic missile role. 

PEELS is a terminal endgame lethality simulation, which models hit-to-kill (HTK) 
and fragment kill of ballistic missiles from Kinetic Energy Weapons (KEW) attack. 

PEELS predicts damage, probability of kill (Pk), and submunitions kill fraction dependent 
on target design using lethality criteria developed for a number of threats. The design fa¬ 
cilitates force-on-force systems studies where six-degree-of-freedom (6-DOP) fly out mod¬ 
els predict endgame state vectors. 

Algorithms in the PEELS code also allow parametric investigations of lethality 
through variation of endgame variables such as miss distance, crossing angle, warhead 
burst point, intercept altitude, target and interceptor velocity and angles of attack, target 
aim point, and fragment pattern density. The model evaluates HTK and fragment kill ef¬ 
fects against both high explosive (HE) and chemical unitary warheads, and against both HE 
and chemical submunitions warheads. Primary applications areas include: Parametric Eea- 
sibility Analysis Studies—^warhead concept evaluation; System Assessment—scenario spe¬ 
cific lethality coupled with a 6-DOP model; Pre-Test Analysis—help define test matrix— 
identify stressing cases to be tested. 

Documentation includes: An Executive Summary and an eight-volume set of 
documentation. Volume 0, Executive Summary; Volume I, PEELS Technical Manual (Al¬ 
gorithm and model detail); Volume 2, PEELS Targets Manual and Volume 3, TMD Lethal¬ 
ity Criteria (Information describing the geometric, structural, and payload design of the tar¬ 
gets and lethality criteria); Volume 4, PEELS Validation and Verification Manual; Volume 
5, User’s Manual; Volume 6, Accreditation Report; Volume 7; Maintenance and Distribu¬ 
tion Manual; Volume 8, Source Code Manual. 
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PEELS runs on SGI ETNIX based system, VAX, or PC systems, and programmed in 
PORTRAN-77. PEELS operates in two basie modes of operation: 1) the Database Mode 
and 2) the Direct Mode. The Database Mode uses a pre-proeessor to ereate databases ae- 
eessed during exeeution, for rapid system level applieations where lethality is only one part 
of many eomputations. The Direct Mode runs if the target model does not exhibit axial- 
symmetry or the desired fragment type is not in the database. This typieally requires four 
to five times more eomputer proeessing time. 

Database Mode requires a series of databases aeeessed during exeeution. These 
eontain information about the solid geometry of the target as well as parameters assoeiated 
with fragment lethality. An Index Database eomposed of shot-line intersection with inter¬ 
nal target components maintains data considered critical to target lethality. The index da¬ 
tabase points to a Lethality Database. The lethality database maintains empirical data. 

This contains information that defines the minimum fragment material and energy charac¬ 
teristics necessary to achieve lethality along each shot line. 
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APPENDIX J - THE DOMAIN METADATA REPOSITORY REGISTRY (DMRR) 


A. COMMON DMRR METADATA FACILITIES 

From [IS079c]^^^ we extract the metadata facilities. This is a critical component 
for achieving Registry interoperability with the domain, inter-domain integration, and at the 
enterprise level. The common DMRR facilities illustrated in Figure J-1 support adminis¬ 
tered items, identified once within the registry. The named administered items have at least 
one context, maybe more, and within each context, names and definitions specified in one 
or more languages. The administered items have classification in zero or more classifica¬ 
tion schemes [IS079c]. 



Figure J-1. BMDS DMRR Common Facilities (From [IS079c]) 


© International Organization for Standardization (ISO). This material is reproduced from ISO/IEC 11179-3, Second Edition, 2003- 
02-15 with permission of the American National Standards Institute on behalf of ISO. No part of this material may be copied or repro¬ 
duced in any form, electronic retrieval system or otherwise or made available on the Internet, a public network by satellite or otherwise 
without the prior written consent of the American National Standards Institute, 25 West 43'“' Street, New York, NY, 10036. 
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B, 


DMRR TYPES OF ADMINISTERED 


[ISO079c] specifies and describes the types of Administered Items^*’^ listed in Fig¬ 
ure J-2, and allows definition of additional types of Administered Items. 



Figure J-2. BMDS DMRR Types of Administered Items (From [IS079c]) 


An Administered Item may be one of the types listed in Figure J-2. Each instance of the Administered Item encapsulates its own 
Administration Record. An Administered Item is submitted by an Organization represented by the relationship submission in Figure J-4. 
A Registration Authority registers Administered Item represented by the relationship registration in Figure J-4. An Organization admin¬ 
isters the Administered Item represented by the relationship stewardship in Figure J-4. A Reference Document may describe zero or 
more Administered Item represented by the relationship reference in Figure J-4. Each instance of the Administered Item through an 
Administered Record has a unique Administered Item Identifier [IS097c]. 
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c. 


DMRR HIGH-LEVEL METAMODEL 


Figure J-2 provides a high-level overview of the central regions of the metamodel. 
[IS079c] 
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Figure J-3. BMDS DMRR High-Level Metamodel Overview (From [IS079c]) 


D. DMRR ADMINISTRATION AND IDENTIFICATION METAMODEL RE¬ 
GION 

The Administration and Identification in Figure J-4 and Figure J-5 supports the ad¬ 
ministrative areas of the Administered items in the DMRR including: 

• Identification and registration of items submitted to the registry, 

• Organizational information and responsible agencies, including Registration Au- 
thorities^^^, 

• Organizational contact information, 

• Supporting documentation, 

• Relationships among administered items [IS079c] 


A Registration Authority is any organization authorized to register metadata. A Registration Authority is a subtype of organization 
and inherits all of its attributes and relationships. An Administered Item has a Registration Authority that is its owner, shown by the 
relationship in Figure J-4. A Registration Authority may register many Administered Items [IS097c]. 
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Figure J-4. BMDS DMRR Admin / Identification Metamodel Region (From [IS079c]) 

E, DMRR CLASSES USED AS COMPOSITE DATATYPES 

[IS097f] prescribes he registration of Administered Items. Figure J-5 shows the 
composite data types used on composite attributes. 


Registration Authority Identifier 

internalional_oode_designator [1 .1] String 
organization_identifier [1 ..1] String 
organization_part_idenbfier [0. 1] . String 
OPI_source [0 .1] String 


Language_ldentification 


!anguage_identifier [1 1] String 
country_identifier [0..1 J String 


Contact 


contact_name [1 1] . String 
contact_btle [0..1) String 
contact_information [1 .1] String 



_ Administration_Record _ 

administered_item_identifier [1. 1 ] ; ltem_ldentifier 
registration_status [1 ..1] . String 
administrative_slatus [1 - 1 ] ; String 
creation_date [1..1] Date 
last_change_date [0. 1] . Date 
effective_date [O .1] . Date 
until_date [0 1J . Date 
change_descrlption (0- 1] ^ String 
administrative_note [0..1 j; String 
explanatory_comment [0. 1] ; String 
unresolved_issue [0..1] String 
origin [O. 1). String 


_ ltem_ldentifier _ 

item_regtstration_authority_jdentirier [1 - .1) Registration_Authority_ldentifier 
data_identifier [1 1 ] . String 
version [1. 1] . String 


Figure J-5. BMDS DMRR Classes / Composite Datatypes in the Administration and 

Identification Region (From [IS079c]) 
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F 


DMRR NAMING AND DEFINITION METAMODEL 


Figure J-6 represents the Naming and Definition Region used to manage the names 
and definitions of administered items and the eontexts for the names. An administered item 
may have many names that will vary based on diseipline, loeality, and teehnology em¬ 
ployed [IS079e]. [fSOVQd]^^"^ provides rules and guidelines for the formulation of data 
definitions, and [IS079e] provides naming and identifieation prineiples for administered 
items within a eontext. 



Figure J-6. BMDS DMRR Naming and Metamodel Region (From [IS079o]) 
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G. 


DMRR CLASSIFICATION METAMODEL REGION 


The Classification Region illustrated in Figure J-7 provides a facility to register and 
administer Classification Schemes^^^ and their constituent Classification Scheme Items. In 
addition, a Classification Scheme may classify Administered Items within the registry. 
Some Classification Schemes will be more applicable to classifying objects in the real 
world than classifying metadata objects in the registry [IS079c]. 



Figure J-7. BMDS DMRR Classification Metamodel Region (From [IS079c]) 


A classification scheme may be a taxonomy, a network, an ontology, or any other terminological system; or the classification scheme 
may consist of a list of controlled vocabulary words and terms. A classification scheme is a sub-type of Administered Item, inherting its 
attributes and relationships, supporting identification, naming, definition, and classification functions [IS079c]. 
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H. DMRR DATA ELEMENT CONCEPT METAMODEL REGION 


The data element concept region illustrated in Figure J-8 maintains the information 
on the concepts upon which the data elements are developed. The metadata objects in this 
region concentrate on semantic concepts. The metadata objects in this region are object 
classes and properties, combined to form Data Element Concepts. The concepts are inde¬ 
pendent of any internal or external physical representation. [IS079c]. A Data Element 
Concept may be represented in the form of a data element, described independently of a 
particular representation, with zero or one object class and zero or one property. A prop- 
erty^^^ is a characteristic common to all members of an object class. 



Figure J-8. BMDS DMRR Conceptual Metamodel Region (From [IS079c]) 


A property may be any feature that humans naturally use to distinguish one individual object from another. It is the human percep¬ 
tion of a single characteristic of an object class in the real world. It is conceptual without an associated means of representation for 
communication [IS079c]. 
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I. DMRR CONCEPTUAL AND VALUE DOMAIN METAMODEL REGION 


The region of the metamodel in Figure J-9 supports the administration of the Con¬ 
ceptual Domains and Value Domains, viewed as logieal eode sets and physieal eode sets. 
Coneeptual Domains support Data Element^^^ Concepts^^^, and Value Domains support 
Data Elements as illustrated in Figure J-9. A Coneeptual Domain is a set of Value Mean¬ 
ings, whieh may be expressed or enumerated by a deseription, and as an Administered Item 
maintains Administrative Record information, allowing identifieation, naming, definition, 
and optional elassifieation within a Classification Schema [IS079o]. 

A Coneeptual Domain may be assoeiated with other Conceptual Domains through a 
Conceptual Domain Relationship, either as a eomposition with other Coneeptual Domains, 
or as a eomponent member of a larger Coneeptual Domain. A Coneeptual Domain may 
speeify a eonstraint sueh as a linear measure as its dimensionality. A speeified dimension¬ 
ality requires that any Value Domain based on the Coneeptual Domain speeify a Unit of 
Measure^^^. An Enumerated Coneeptual Domain, a sub-type of Coneeptual Domain may 
eontain a finite number of enumerated notions (e.g., eodes representing the names of 
states). Conversely, a Non-enumerated Coneeptual Domain laeks a finite set of Value 
Meanings. As a sub-type of the Coneeptual Domain, both the Enumerated Coneeptual 
Domain and Non-enumerated Coneeptual Domain inherit the attributes and relationships of 
the former [IS079e]. 

In an Enumerated Coneeptual Domain, eaeh member has a Value Meaning, in the 
registry distinguishing it from other members, and is independent of their representation in 
any eorresponding Value Domain . A speeifie Value Meaning may have more than one 


A data element is a basic unit of data of interest to an organization. It is a unit of data defined, identified, represented, and conatins 
permissible values specified by means of a set of attributes [IS079c]. 

Data Element Concepts is a concept represented in the form of a data element, and described independently of any specific represen¬ 
tation. It may have zero or one Object Class and zero or one Property [IS079c]. 
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A Unit of Measure is the unit associated when Data Element values are specified. The Unit of Measure Name designates the unit. 
When specified the unit must maintain consistency with any dimensionality specified in the corresponding Conceptual Domain. Option¬ 
ally, a Unit of Measure may have specifications for the number of decimal spaces supported in the associated Data Element values. The 
precision is a default tha may be overridden for any particular data element [IS079c]. 
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A Value Domain is a key component of a Conceptual Domain, providing representation for the Conceptual Domain. An example of 
a Value Domain is ISO 3166 describing the codes of the representation of countries within a set of seven Value Domains: short English 
name, official English name, short French name, official French name, apha-2 code, apha-3 code, and numeric code [IS079c]. 
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means of representation by Permissible Values^^‘, from a distinct Enumerated Value Do- 
main . A Value Domain may be associated with other Value Domains through a Value 
Domain Relationship, either as a composition with other Conceptual Domains, or as a 
component member of a larger Conceptual Domain, using the value domain relationship 
type description [IS079c]. 



Figure J-9. Conceptual And Value Domain Metamodel Region (From [IS079c]) 


A permissible value is an expression of a Value Meaning within an Enumerated Value Domain and one of a set of values comprising 

an Enumerated Value Domain. Each permissible value is associated with a value meaning [IS079c]. 
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An Enumerated Value Domain is an expression of an explicit set of two or more permissible values, and a Non-Enumerated Value 
Domain is a value domain expressed by a description, specification or rule, rather than a set of permissible values. Both the Enumerated 
Value Domain and the Non-Enumerated Value Domain are a sub-type of the value domain and inherit the attributes and relationships of 
the value domain [lS079c]. 
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J. DMRR DATA ELEMENT METAMODEL REGION 

The Data Element metamodel region, shown in Figure J-10, addresses the admini¬ 
stration of data elements. Data Elements provide the formal representation of information 
(e.g., a faet, a preposition, an observation) about some eonerete or abstraet thing. Data 
Elements are reusable and sharable representations of Data Element Coneepts. When a 
Data Element Coneept reeeives a value, a Data Element forms. A Data Element is the as- 
soeiation among a Data Element Coneept, a Value Domain, and a Representation Class 
[IS079o]. 



Figure J-10. BMDS DMRR Data Element Metamodel Region (From [IS079o]) 
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The Data Element to Representation Class assoeiation may oeeur directly or by way 
of the Value domain. Registration of a Data Element as an Administered Item requires as¬ 
sociation with a Data Element Concept and a Value Domain. A representation class quali¬ 
fier, if specified, qualifies the name of the data element. Data Element Precision specifies 
the number of decimal spaces permitted by any associated data element values. Unless 
specified differently, the unit of measure precision from the associated Value Domain ap- 
plies. A data element may also have Data Element Examples and a Derivation Rule 
[IS079c]. 

A Data Element may have associations with several Value Domains, resulting in a 
different Data Element Concept for each association. The Value Domain provides repre¬ 
sentation, but no links to the Data Element Concept values, or what the values mean. A 

T'7C 

Value Domain may associate with multiple Data Elements. The Representation Class is 
the Classification Scheme for representation, supporting a set of classes to distinguish 
among the various elements in the registry. Objectives of the Representation Class include 
providing a complete, discrete set of high-level definitions for data element/value domain 
categorization, supporting the integration of business rules for applications. Representa¬ 
tional Class use enhances semantic control over contents of value domains, supporting the 
development of rules for representation classes allowing the enforcement of content within 
and among domain values [IS079c]. [IS079e] and [IS079c] provide an informational list 
or representational class terms. 

K. DMRR CONSOLIDATED METAMODEL 

A consolidated metamodel in Eigure J-11 illustrates the combined Data Element 
Concept, Data Element, Conceptual Domain, and Value Domain of the [IS079c] Model. 
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Data Element Examples provide representative samples of the Data Element [IS079c]. 
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A Derivation Rule specifies a derivation for the Data Element [iS079c]. 

375 

A Representation Class is a mechanism for conveying the functional and/or presentational category of an item to a user [iS079c]. 
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Figure J-11. BMDS DMRR Consolidated Metamodel (From [IS079c]) 
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